Re: [Discuss] Publishing pre-release artifacts to repositories

[Michael Luckey <ad...@gmail.com>]

Thanks Ahmet for calling out to the airflow folks. I believe, I am able to
follow their argument. So from my point of view I do not have an issue with
apache policy. But honestly still trying to wrap my head around Roberts
concern with rebuilding/resigning. Currently, our actual release is only a
tag on source repo and promoting artefacts. Do not yet understand how that
needs to change to get PyPi included.

On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <al...@google.com> wrote:

> Michael, Max and other folks who are concerned about the compatibility
> with the apache release policy. Does the information in this thread
> sufficiently address your concerns? Especially the part where, the rc
> artifacts will be protected by a flag (i.e. --pre) from general consumption.
>
> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay <al...@google.com> wrote:
>> >
>> > This conversation get quite Python centric. Is there a similar need for
>> Java?
>>
>> I think Java is already covered. Go is a different story (but the even
>> versioning and releasing is being worked out).
>>
>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw <ro...@google.com>
>> wrote:
>> >>
>> >> If we can, by the apache guidelines, post RCs to pypy that is
>> >> definitely the way to go. (Note that test.pypi is for developing
>> >> against the pypi interface, not for pushing anything real.) The caveat
>> >> about naming these with rcN in the version number still applies
>> >> (that's how pypi guards them against non-explicit installs).
>> >
>> > Related to the caveat, I believe this can be easily scripted or even
>> made part of the travis/wheels pipeline to take the release branch, edit
>> the version string in place to add rc, and build the necessary files.
>>
>> Yes. But the resulting artifacts would have to be rebuilt (and
>> re-signed) without the version edit for the actual release. (Well, we
>> could possibly edit the artifacts rather than rebuild them.) And
>> pushing un-edited ones early would be really bad. (It's the classic
>> tension of whether a pre-release should be marked internally or
>> externally, re-publishing a new set of bits for the actual release or
>> re-using version numbers for different sets of bits. Pypi does one,
>> apache does another...)
>>
>> >> The advantage is that a user can do "pip install --pre apache-beam" to
>> >> get the latest rc rather than "pip install
>> >>
>> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"
>> >>
>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada <pa...@google.com>
>> wrote:
>> >> >
>> >> > Aw that's interesting!
>> >> >
>> >> > I think, with these considerations, I am only marginally more
>> inclined towards publishing to test.pypi. That would make me a +0.9 on
>> publishing RCs to the main pip repo then.
>> >> >
>> >> > Thanks for doing the research Ahmet. :)
>> >> > Best
>> >> > -P
>> >> >
>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <al...@google.com>
>> wrote:
>> >> >>
>> >> >> I asked to Airflow folks about this. See [1] for the full response
>> and a link to one of their RC emails. To summarize their position
>> (specifically for pypi) is: Unless a user does something explicit (such as
>> using a flag, or explicitly requesting an rc release), pip install will not
>> serve RC binaries. And that is compatible with RC section of
>> http://www.apache.org/legal/release-policy.html#release-types
>> >> >>
>> >> >> Ahmet
>> >> >>
>> >> >> [1]
>> https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E
>> >> >>
>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <al...@google.com>
>> wrote:
>> >> >>>
>> >> >>> The incremental value of publishing python artifacts to a separate
>> place but not to actual pypi listing will be low. Users can already
>> download RC artifacts, or even pip install from http location directly. I
>> think the incremental value will be low, because for a user or a downstream
>> library to test with Beam RCs using their usual ways will still require
>> them to get other dependencies from the regular pypi listing. That would
>> mean they need to change their setup to test with beam rcs, which is the
>> same state as today. There will be some incremental value of putting them
>> in more obvious places (e.g. pypi test repository). I would rather not
>> complicate the release process for doing this.
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles <ke...@apache.org>
>> wrote:
>> >> >>>>
>> >> >>>> Pip is also able to be pointed at any raw hosted directory for
>> the install, right? So we could publish RCs or snapshots somewhere with
>> more obvious caveats and not interfere with the pypi list of actual
>> releases. Much like the Java snapshots are stored in a separate opt-in
>> repository.
>> >> >>>>
>> >> >>>> Kenn
>> >> >>>>
>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels <
>> mxm@apache.org> wrote:
>> >> >>>>>
>> >> >>>>> > wouldn't that be in conflict with Apache release policy [1] ?
>> >> >>>>> > [1] http://www.apache.org/legal/release-policy.html
>> >> >>>>>
>> >> >>>>> Indeed, advertising pre-release artifacts is against ASF rules.
>> For
>> >> >>>>> example, Flink was asked to remove a link to the Maven snapshot
>> >> >>>>> repository from their download page.
>> >> >>>>>
>> >> >>>>> However, that does not mean we cannot publish Python artifacts.
>> We just
>> >> >>>>> have to clearly mark them for developers only and not advertise
>> them
>> >> >>>>> alongside with the official releases.
>> >> >>>>>
>> >> >>>>> -Max
>> >> >>>>>
>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote:
>> >> >>>>> > Don't we push java artifacts to maven repositories as part of
>> the RC
>> >> >>>>> > process? And completely unvetted snapshots? (Or is this OK
>> because
>> >> >>>>> > they are special opt-in apache-only ones?)
>> >> >>>>> >
>> >> >>>>> > I am generally in favor of the idea, but would like to avoid
>> increased
>> >> >>>>> > toil on the release manager.
>> >> >>>>> >
>> >> >>>>> > One potential hitch I see is that current release process
>> updates the
>> >> >>>>> > versions to x.y.z (no RC or other pre-release indicator in the
>> version
>> >> >>>>> > number) whereas pypi (and other systems) typically expect
>> distinct
>> >> >>>>> > (recognizable) version numbers for each attempt, and only the
>> actual
>> >> >>>>> > final result has the actual final release version.
>> >> >>>>> >
>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay <al...@google.com>
>> wrote:
>> >> >>>>> >>
>> >> >>>>> >> I do not know the answer.I believe this will be similar to
>> sharing the RC artifacts for validation purposes and would not be a formal
>> release by itself. But I am not an expert and I hope others will share
>> their opinions.
>> >> >>>>> >>
>> >> >>>>> >> I quickly searched pypi for apache projects and found at
>> least airflow [1] and libcloud [2] are publishing rc artifacts to pypi. We
>> can reach out to those communities and learn about their processes.
>> >> >>>>> >>
>> >> >>>>> >> Ahmet
>> >> >>>>> >>
>> >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history
>> >> >>>>> >> [2] https://pypi.org/project/apache-libcloud/#history
>> >> >>>>> >>
>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey <
>> adude3141@gmail.com> wrote:
>> >> >>>>> >>>
>> >> >>>>> >>> Hi,
>> >> >>>>> >>>
>> >> >>>>> >>> wouldn't that be in conflict with Apache release policy [1] ?
>> >> >>>>> >>>
>> >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html
>> >> >>>>> >>>
>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold <
>> amyrvold@google.com> wrote:
>> >> >>>>> >>>>
>> >> >>>>> >>>> Great idea. I like the RC candidates to follow as much as
>> the release artifact process as possible.
>> >> >>>>> >>>>
>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay <
>> altay@google.com> wrote:
>> >> >>>>> >>>>>
>> >> >>>>> >>>>> To clarify my proposal, I am proposing publishing to the
>> production pypi repository with an rc tag in the version. And in turn allow
>> users to depend on beam's rc version + all the other regular dependencies
>> users would have directly from pypi.
>> >> >>>>> >>>>>
>> >> >>>>> >>>>> Publishing to test pypi repo would also be helpful if test
>> pypi repo also mirrors other packages that exist in the production pypi
>> repository.
>> >> >>>>> >>>>>
>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada <
>> pabloem@google.com> wrote:
>> >> >>>>> >>>>>>
>> >> >>>>> >>>>>> I think this is a great idea. A way of doing it for
>> python would be by using the test repository for PyPi[1], and that way we
>> would not have to do an official PyPi release, but still would be able to
>> install it with pip (by passing an extra flag), and test.
>> >> >>>>> >>>>>>
>> >> >>>>> >>>>>> In fact, there are some Beam artifacts already in
>> there[2]. At some point I looked into this, but couldn't figure out who has
>> access/the password for it.
>> >> >>>>> >>>>>
>> >> >>>>> >>>>>
>> >> >>>>> >>>>> I also don't know who owns beam package in test pypi repo.
>> Does anybody know?
>> >> >>>>> >>>>>
>> >> >>>>> >>>>>>
>> >> >>>>> >>>>>>
>> >> >>>>> >>>>>> In short: +1, and I would suggest using the test PyPi
>> repo to avoid publishing to the main PyPi repo.
>> >> >>>>> >>>>>> Best
>> >> >>>>> >>>>>> -P.
>> >> >>>>> >>>>>>
>> >> >>>>> >>>>>> [1] https://test.pypi.org/
>> >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/
>> >> >>>>> >>>>>>
>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay <
>> altay@google.com> wrote:
>> >> >>>>> >>>>>>>
>> >> >>>>> >>>>>>> Hi all,
>> >> >>>>> >>>>>>>
>> >> >>>>> >>>>>>> What do you think about the idea of publishing
>> pre-release artifacts as part of the RC emails?
>> >> >>>>> >>>>>>>
>> >> >>>>> >>>>>>> For Python this would translate into publishing the same
>> artifacts from RC email with a version like "2.X.0rcY" to pypi. I do not
>> know, but I am guessing we can do a similar thing with Maven central for
>> Java artifacts as well.
>> >> >>>>> >>>>>>>
>> >> >>>>> >>>>>>> Advantages would be:
>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their own purposes
>> using the same exact process they will normally use.
>> >> >>>>> >>>>>>>   - Enable early-adaptors to start using RC releases
>> early on in the release cycle if that is what they would like to do. This
>> will in turn reduce time pressure on some releases. Especially for cases
>> like someone needs a release to be finalized for an upcoming event.
>> >> >>>>> >>>>>>>
>> >> >>>>> >>>>>>> There will also be disadvantages, some I could think of:
>> >> >>>>> >>>>>>> - Users could request support for RC artifacts.
>> Hopefully in the form of feedback for us to improve the release. But it
>> could also be in the form of folks using RC artifacts for production for a
>> long time.
>> >> >>>>> >>>>>>> - It will add toil to the current release process, there
>> will be one more step for each RC. I think for python this will be a small
>> step but nevertheless it will be additional work.
>> >> >>>>> >>>>>>>
>> >> >>>>> >>>>>>> For an example of this, you can take a look at
>> tensorflow releases. For 1.13 there were 3 pre-releases [1].
>> >> >>>>> >>>>>>>
>> >> >>>>> >>>>>>> Ahmet
>> >> >>>>> >>>>>>>
>> >> >>>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history
>>
>

Re: [Discuss] Publishing pre-release artifacts to repositories

[Ahmet Altay <al...@google.com>]

Thank you all for feedback. I filed BEAM-7242 for myself to update the
release process with sufficient tooling to add support for this. I am
unlikely to be able to do it for the current release but hopefully I will
work on it soon.

*From: *Michael Luckey <ad...@gmail.com>
*Date: *Mon, May 6, 2019 at 4:48 PM
*To: * <de...@beam.apache.org>

Thanks Ahmet for the time you put into this. AFAIU Roberts proposal
> resolves my concerns.
>
> On Mon, May 6, 2019 at 8:01 PM Ahmet Altay <al...@google.com> wrote:
>
>> Thank you Max. Michael, does the current state of the proposal address
>> your concerns?
>>
>> *From: *Maximilian Michels <mx...@apache.org>
>> *Date: *Mon, May 6, 2019 at 8:32 AM
>> *To: * <de...@beam.apache.org>
>>
>> Thanks for the link to the distribution guideline thread. The proposed
>>> process sounds good to me.
>>>
>>> Concerning Michael's point, I think we should follow the same procedure
>>> as we do for releasing to Maven Central, i.e. create a staging
>>> repository for validation which then gets released.
>>>
>>> The easiest way to do that, is to directly use the artifacts that we
>>> create during the release process:
>>>
>>>    pip install https://dist.apache.org/repos/dist\
>>>                /dev/beam/<version>/python
>>>
>>> We wouldn't have to built a second artifact or change a version number.
>>> However, I understand if we want the additional convenience of:
>>>
>>>    pip install --pre apache_beam
>>>
>>> As Robert mentioned, this would just be the additional step to rename
>>> the version number and push the RC package to PyPi. Not necessary in my
>>> opinion but I don't mind.
>>>
>>
>> Value proposition is for downstream library providers. They can update
>> their setup.py file to test beam rc in combination with rest of their other
>> dependencies and test that combination with ease. I agree it is not
>> necessary for us or any other party that just wants to test beam packages
>> and its direct dependencies in isolation.
>>
>>
>>>
>>> Thanks,
>>> Max
>>>
>>> On 02.05.19 19:43, Ahmet Altay wrote:
>>> >
>>> >
>>> > On Thu, May 2, 2019 at 9:29 AM Robert Bradshaw <robertwb@google.com
>>> > <ma...@google.com>> wrote:
>>> >
>>> >     On Thu, May 2, 2019 at 6:03 PM Michael Luckey <adude3141@gmail.com
>>> >     <ma...@gmail.com>> wrote:
>>> >      >
>>> >      > Yes, I understood this. But I m personally more paranoid about
>>> >     releasing.
>>> >      >
>>> >      > So formally vote (and corresponding testing) was done on rc. If
>>> >     we rebuild and resign, wouldn't that mean we also need to revote?
>>> >
>>> >     Yeah, that's the sticking point. I suppose we could build the
>>> packages
>>> >     with rc tags, push them to pypi, and also build them without rc
>>> tags,
>>> >     and push those (and the full source tarball, which doesn't have an
>>> rc
>>> >     tag either) to svn, and have the vote officially cover what's in
>>> svn
>>> >     but the rc ones are just for convenience.
>>> >
>>> >
>>> > I prefer this way of double building. I think modifying the release
>>> > process as in your initial proposal would be risky. In that proposal,
>>> > what we test for validations would be different than what we push out
>>> as
>>> > final releases. The modification I see would be:
>>> >
>>> > 2c) Get the artifacts from svn, un-package, change the version to
>>> > include the rc number, and re-package and push. (This is very similar
>>> to
>>> > the process we use for building the wheel files today, with the
>>> > exception of changing the version part.)
>>> >
>>> >     (But, given that I can "pip
>>> >     install https::svn.apache.org/path/to/tarball
>>> >     <http://svn.apache.org/path/to/tarball>" it'd primarily have
>>> >     value for others doing "pip install --pre".)
>>> >
>>> >
>>> >     This is regardless of whether is OK per apache to publish such
>>> binary
>>> >     blobs to a third party place (though IMHO it follows the intent of
>>> the
>>> >     release process).
>>> >
>>> >
>>> > This is also my understanding. What is being proposed here aligns with
>>> > what Kenn found out in the other threads.
>>> >
>>> >
>>> >      > If I understand correctly, there will be some changed version
>>> >     string in distributed sources (setup.py?). So there is some binary
>>> >     difference. And just talking about me, doing that repackaging I
>>> >     would certainly mess it up and package some unwanted changes.
>>> >
>>> >     We definitely would not want this to be a manual step--I wouldn't
>>> >     trust myself :).
>>> >
>>> >
>>> >      > On Thu, May 2, 2019 at 5:43 PM Robert Bradshaw
>>> >     <robertwb@google.com <ma...@google.com>> wrote:
>>> >      >>
>>> >      >> On Thu, May 2, 2019 at 5:24 PM Michael Luckey
>>> >     <adude3141@gmail.com <ma...@gmail.com>> wrote:
>>> >      >> >
>>> >      >> > Thanks Ahmet for calling out to the airflow folks. I believe,
>>> >     I am able to follow their argument. So from my point of view I do
>>> >     not have an issue with apache policy. But honestly still trying to
>>> >     wrap my head around Roberts concern with rebuilding/resigning.
>>> >     Currently, our actual release is only a tag on source repo and
>>> >     promoting artefacts. Do not yet understand how that needs to change
>>> >     to get PyPi included.
>>> >      >>
>>> >      >> It's not a big change, but let me clarify.
>>> >      >>
>>> >      >> Currently our release preparation goes something like this:
>>> >      >>
>>> >      >> 1) Check out the repo, update the versions to 2.x, build and
>>> >     sign the artifacts.
>>> >      >> 2) Announce these artifacts as rcN
>>> >      >> 2a) Push the artifacts to SVN dev/...
>>> >      >> 2b) Push artifacts to the apache maven repository.
>>> >      >> 3) Depending on vote, go back to step (1) or forward to step
>>> (4).
>>> >      >> 4) Copy these artifacts as the actual release.
>>> >      >>
>>> >      >> Now if we just try to add (2c) Push these artifacts to Pypi,
>>> it will
>>> >      >> be treated (by pypi's tooling, anyone who downloads the
>>> tarball,
>>> >     ...)
>>> >      >> as an actual release. You also can't re-push a tarball with
>>> the same
>>> >      >> name and different contents (the idea being that named releases
>>> >     should
>>> >      >> never change). So we'd need to change step (1) to update the
>>> version
>>> >      >> to 2.x.rcN *and* add a step in (4) to update the version to 2.x
>>> >     (no rc
>>> >      >> suffix), rebuild, resign before publishing.
>>> >      >>
>>> >      >> As mentioned, possibly the rcN suffix could be part of the
>>> building
>>> >      >> step for Python.
>>> >      >>
>>> >      >> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <altay@google.com
>>> >     <ma...@google.com>> wrote:
>>> >      >> >>
>>> >      >> >> Michael, Max and other folks who are concerned about the
>>> >     compatibility with the apache release policy. Does the information
>>> >     in this thread sufficiently address your concerns? Especially the
>>> >     part where, the rc artifacts will be protected by a flag (i.e.
>>> >     --pre) from general consumption.
>>> >      >> >>
>>> >      >> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw
>>> >     <robertwb@google.com <ma...@google.com>> wrote:
>>> >      >> >>>
>>> >      >> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay
>>> >     <altay@google.com <ma...@google.com>> wrote:
>>> >      >> >>> >
>>> >      >> >>> > This conversation get quite Python centric. Is there a
>>> >     similar need for Java?
>>> >      >> >>>
>>> >      >> >>> I think Java is already covered. Go is a different story
>>> >     (but the even
>>> >      >> >>> versioning and releasing is being worked out).
>>> >      >> >>>
>>> >      >> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw
>>> >     <robertwb@google.com <ma...@google.com>> wrote:
>>> >      >> >>> >>
>>> >      >> >>> >> If we can, by the apache guidelines, post RCs to pypy
>>> that is
>>> >      >> >>> >> definitely the way to go. (Note that test.pypi is for
>>> >     developing
>>> >      >> >>> >> against the pypi interface, not for pushing anything
>>> >     real.) The caveat
>>> >      >> >>> >> about naming these with rcN in the version number still
>>> >     applies
>>> >      >> >>> >> (that's how pypi guards them against non-explicit
>>> installs).
>>> >      >> >>> >
>>> >      >> >>> > Related to the caveat, I believe this can be easily
>>> >     scripted or even made part of the travis/wheels pipeline to take
>>> the
>>> >     release branch, edit the version string in place to add rc, and
>>> >     build the necessary files.
>>> >      >> >>>
>>> >      >> >>> Yes. But the resulting artifacts would have to be rebuilt
>>> (and
>>> >      >> >>> re-signed) without the version edit for the actual release.
>>> >     (Well, we
>>> >      >> >>> could possibly edit the artifacts rather than rebuild
>>> them.) And
>>> >      >> >>> pushing un-edited ones early would be really bad. (It's the
>>> >     classic
>>> >      >> >>> tension of whether a pre-release should be marked
>>> internally or
>>> >      >> >>> externally, re-publishing a new set of bits for the actual
>>> >     release or
>>> >      >> >>> re-using version numbers for different sets of bits. Pypi
>>> >     does one,
>>> >      >> >>> apache does another...)
>>> >      >> >>>
>>> >      >> >>> >> The advantage is that a user can do "pip install --pre
>>> >     apache-beam" to
>>> >      >> >>> >> get the latest rc rather than "pip install
>>> >      >> >>> >>
>>> >
>>> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"
>>> >      >> >>> >>
>>> >      >> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada
>>> >     <pabloem@google.com <ma...@google.com>> wrote:
>>> >      >> >>> >> >
>>> >      >> >>> >> > Aw that's interesting!
>>> >      >> >>> >> >
>>> >      >> >>> >> > I think, with these considerations, I am only
>>> >     marginally more inclined towards publishing to test.pypi. That
>>> would
>>> >     make me a +0.9 on publishing RCs to the main pip repo then.
>>> >      >> >>> >> >
>>> >      >> >>> >> > Thanks for doing the research Ahmet. :)
>>> >      >> >>> >> > Best
>>> >      >> >>> >> > -P
>>> >      >> >>> >> >
>>> >      >> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay
>>> >     <altay@google.com <ma...@google.com>> wrote:
>>> >      >> >>> >> >>
>>> >      >> >>> >> >> I asked to Airflow folks about this. See [1] for the
>>> >     full response and a link to one of their RC emails. To summarize
>>> >     their position (specifically for pypi) is: Unless a user does
>>> >     something explicit (such as using a flag, or explicitly requesting
>>> >     an rc release), pip install will not serve RC binaries. And that is
>>> >     compatible with RC section of
>>> >     http://www.apache.org/legal/release-policy.html#release-types
>>> >      >> >>> >> >>
>>> >      >> >>> >> >> Ahmet
>>> >      >> >>> >> >>
>>> >      >> >>> >> >> [1]
>>> >
>>> https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E
>>> >      >> >>> >> >>
>>> >      >> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay
>>> >     <altay@google.com <ma...@google.com>> wrote:
>>> >      >> >>> >> >>>
>>> >      >> >>> >> >>> The incremental value of publishing python artifacts
>>> >     to a separate place but not to actual pypi listing will be low.
>>> >     Users can already download RC artifacts, or even pip install from
>>> >     http location directly. I think the incremental value will be low,
>>> >     because for a user or a downstream library to test with Beam RCs
>>> >     using their usual ways will still require them to get other
>>> >     dependencies from the regular pypi listing. That would mean they
>>> >     need to change their setup to test with beam rcs, which is the same
>>> >     state as today. There will be some incremental value of putting
>>> them
>>> >     in more obvious places (e.g. pypi test repository). I would rather
>>> >     not complicate the release process for doing this.
>>> >      >> >>> >> >>>
>>> >      >> >>> >> >>>
>>> >      >> >>> >> >>>
>>> >      >> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles
>>> >     <kenn@apache.org <ma...@apache.org>> wrote:
>>> >      >> >>> >> >>>>
>>> >      >> >>> >> >>>> Pip is also able to be pointed at any raw hosted
>>> >     directory for the install, right? So we could publish RCs or
>>> >     snapshots somewhere with more obvious caveats and not interfere
>>> with
>>> >     the pypi list of actual releases. Much like the Java snapshots are
>>> >     stored in a separate opt-in repository.
>>> >      >> >>> >> >>>>
>>> >      >> >>> >> >>>> Kenn
>>> >      >> >>> >> >>>>
>>> >      >> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels
>>> >     <mxm@apache.org <ma...@apache.org>> wrote:
>>> >      >> >>> >> >>>>>
>>> >      >> >>> >> >>>>> > wouldn't that be in conflict with Apache release
>>> >     policy [1] ?
>>> >      >> >>> >> >>>>> > [1]
>>> http://www.apache.org/legal/release-policy.html
>>> >      >> >>> >> >>>>>
>>> >      >> >>> >> >>>>> Indeed, advertising pre-release artifacts is
>>> >     against ASF rules. For
>>> >      >> >>> >> >>>>> example, Flink was asked to remove a link to the
>>> >     Maven snapshot
>>> >      >> >>> >> >>>>> repository from their download page.
>>> >      >> >>> >> >>>>>
>>> >      >> >>> >> >>>>> However, that does not mean we cannot publish
>>> >     Python artifacts. We just
>>> >      >> >>> >> >>>>> have to clearly mark them for developers only and
>>> >     not advertise them
>>> >      >> >>> >> >>>>> alongside with the official releases.
>>> >      >> >>> >> >>>>>
>>> >      >> >>> >> >>>>> -Max
>>> >      >> >>> >> >>>>>
>>> >      >> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote:
>>> >      >> >>> >> >>>>> > Don't we push java artifacts to maven
>>> >     repositories as part of the RC
>>> >      >> >>> >> >>>>> > process? And completely unvetted snapshots? (Or
>>> >     is this OK because
>>> >      >> >>> >> >>>>> > they are special opt-in apache-only ones?)
>>> >      >> >>> >> >>>>> >
>>> >      >> >>> >> >>>>> > I am generally in favor of the idea, but would
>>> >     like to avoid increased
>>> >      >> >>> >> >>>>> > toil on the release manager.
>>> >      >> >>> >> >>>>> >
>>> >      >> >>> >> >>>>> > One potential hitch I see is that current
>>> release
>>> >     process updates the
>>> >      >> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release
>>> >     indicator in the version
>>> >      >> >>> >> >>>>> > number) whereas pypi (and other systems)
>>> >     typically expect distinct
>>> >      >> >>> >> >>>>> > (recognizable) version numbers for each attempt,
>>> >     and only the actual
>>> >      >> >>> >> >>>>> > final result has the actual final release
>>> version.
>>> >      >> >>> >> >>>>> >
>>> >      >> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay
>>> >     <altay@google.com <ma...@google.com>> wrote:
>>> >      >> >>> >> >>>>> >>
>>> >      >> >>> >> >>>>> >> I do not know the answer.I believe this will be
>>> >     similar to sharing the RC artifacts for validation purposes and
>>> >     would not be a formal release by itself. But I am not an expert and
>>> >     I hope others will share their opinions.
>>> >      >> >>> >> >>>>> >>
>>> >      >> >>> >> >>>>> >> I quickly searched pypi for apache projects and
>>> >     found at least airflow [1] and libcloud [2] are publishing rc
>>> >     artifacts to pypi. We can reach out to those communities and learn
>>> >     about their processes.
>>> >      >> >>> >> >>>>> >>
>>> >      >> >>> >> >>>>> >> Ahmet
>>> >      >> >>> >> >>>>> >>
>>> >      >> >>> >> >>>>> >> [1]
>>> https://pypi.org/project/apache-airflow/#history
>>> >      >> >>> >> >>>>> >> [2]
>>> >     https://pypi.org/project/apache-libcloud/#history
>>> >      >> >>> >> >>>>> >>
>>> >      >> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey
>>> >     <adude3141@gmail.com <ma...@gmail.com>> wrote:
>>> >      >> >>> >> >>>>> >>>
>>> >      >> >>> >> >>>>> >>> Hi,
>>> >      >> >>> >> >>>>> >>>
>>> >      >> >>> >> >>>>> >>> wouldn't that be in conflict with Apache
>>> >     release policy [1] ?
>>> >      >> >>> >> >>>>> >>>
>>> >      >> >>> >> >>>>> >>> [1]
>>> http://www.apache.org/legal/release-policy.html
>>> >      >> >>> >> >>>>> >>>
>>> >      >> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold
>>> >     <amyrvold@google.com <ma...@google.com>> wrote:
>>> >      >> >>> >> >>>>> >>>>
>>> >      >> >>> >> >>>>> >>>> Great idea. I like the RC candidates to
>>> follow
>>> >     as much as the release artifact process as possible.
>>> >      >> >>> >> >>>>> >>>>
>>> >      >> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay
>>> >     <altay@google.com <ma...@google.com>> wrote:
>>> >      >> >>> >> >>>>> >>>>>
>>> >      >> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing
>>> >     publishing to the production pypi repository with an rc tag in the
>>> >     version. And in turn allow users to depend on beam's rc version +
>>> >     all the other regular dependencies users would have directly from
>>> pypi.
>>> >      >> >>> >> >>>>> >>>>>
>>> >      >> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be
>>> >     helpful if test pypi repo also mirrors other packages that exist in
>>> >     the production pypi repository.
>>> >      >> >>> >> >>>>> >>>>>
>>> >      >> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo
>>> Estrada
>>> >     <pabloem@google.com <ma...@google.com>> wrote:
>>> >      >> >>> >> >>>>> >>>>>>
>>> >      >> >>> >> >>>>> >>>>>> I think this is a great idea. A way of
>>> doing
>>> >     it for python would be by using the test repository for PyPi[1],
>>> and
>>> >     that way we would not have to do an official PyPi release, but
>>> still
>>> >     would be able to install it with pip (by passing an extra flag),
>>> and
>>> >     test.
>>> >      >> >>> >> >>>>> >>>>>>
>>> >      >> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts
>>> >     already in there[2]. At some point I looked into this, but couldn't
>>> >     figure out who has access/the password for it.
>>> >      >> >>> >> >>>>> >>>>>
>>> >      >> >>> >> >>>>> >>>>>
>>> >      >> >>> >> >>>>> >>>>> I also don't know who owns beam package in
>>> >     test pypi repo. Does anybody know?
>>> >      >> >>> >> >>>>> >>>>>
>>> >      >> >>> >> >>>>> >>>>>>
>>> >      >> >>> >> >>>>> >>>>>>
>>> >      >> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the
>>> >     test PyPi repo to avoid publishing to the main PyPi repo.
>>> >      >> >>> >> >>>>> >>>>>> Best
>>> >      >> >>> >> >>>>> >>>>>> -P.
>>> >      >> >>> >> >>>>> >>>>>>
>>> >      >> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/
>>> >      >> >>> >> >>>>> >>>>>> [2]
>>> https://test.pypi.org/project/apache-beam/
>>> >      >> >>> >> >>>>> >>>>>>
>>> >      >> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay
>>> >     <altay@google.com <ma...@google.com>> wrote:
>>> >      >> >>> >> >>>>> >>>>>>>
>>> >      >> >>> >> >>>>> >>>>>>> Hi all,
>>> >      >> >>> >> >>>>> >>>>>>>
>>> >      >> >>> >> >>>>> >>>>>>> What do you think about the idea of
>>> >     publishing pre-release artifacts as part of the RC emails?
>>> >      >> >>> >> >>>>> >>>>>>>
>>> >      >> >>> >> >>>>> >>>>>>> For Python this would translate into
>>> >     publishing the same artifacts from RC email with a version like
>>> >     "2.X.0rcY" to pypi. I do not know, but I am guessing we can do a
>>> >     similar thing with Maven central for Java artifacts as well.
>>> >      >> >>> >> >>>>> >>>>>>>
>>> >      >> >>> >> >>>>> >>>>>>> Advantages would be:
>>> >      >> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for
>>> their
>>> >     own purposes using the same exact process they will normally use.
>>> >      >> >>> >> >>>>> >>>>>>>   - Enable early-adaptors to start using
>>> RC
>>> >     releases early on in the release cycle if that is what they would
>>> >     like to do. This will in turn reduce time pressure on some
>>> releases.
>>> >     Especially for cases like someone needs a release to be finalized
>>> >     for an upcoming event.
>>> >      >> >>> >> >>>>> >>>>>>>
>>> >      >> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I
>>> >     could think of:
>>> >      >> >>> >> >>>>> >>>>>>> - Users could request support for RC
>>> >     artifacts. Hopefully in the form of feedback for us to improve the
>>> >     release. But it could also be in the form of folks using RC
>>> >     artifacts for production for a long time.
>>> >      >> >>> >> >>>>> >>>>>>> - It will add toil to the current release
>>> >     process, there will be one more step for each RC. I think for
>>> python
>>> >     this will be a small step but nevertheless it will be additional
>>> work.
>>> >      >> >>> >> >>>>> >>>>>>>
>>> >      >> >>> >> >>>>> >>>>>>> For an example of this, you can take a
>>> look
>>> >     at tensorflow releases. For 1.13 there were 3 pre-releases [1].
>>> >      >> >>> >> >>>>> >>>>>>>
>>> >      >> >>> >> >>>>> >>>>>>> Ahmet
>>> >      >> >>> >> >>>>> >>>>>>>
>>> >      >> >>> >> >>>>> >>>>>>> [1]
>>> >     https://pypi.org/project/tensorflow/#history
>>> >
>>>
>>

Re: [Discuss] Publishing pre-release artifacts to repositories

[Michael Luckey <ad...@gmail.com>]

Thanks Ahmet for the time you put into this. AFAIU Roberts proposal
resolves my concerns.

On Mon, May 6, 2019 at 8:01 PM Ahmet Altay <al...@google.com> wrote:

> Thank you Max. Michael, does the current state of the proposal address
> your concerns?
>
> *From: *Maximilian Michels <mx...@apache.org>
> *Date: *Mon, May 6, 2019 at 8:32 AM
> *To: * <de...@beam.apache.org>
>
> Thanks for the link to the distribution guideline thread. The proposed
>> process sounds good to me.
>>
>> Concerning Michael's point, I think we should follow the same procedure
>> as we do for releasing to Maven Central, i.e. create a staging
>> repository for validation which then gets released.
>>
>> The easiest way to do that, is to directly use the artifacts that we
>> create during the release process:
>>
>>    pip install https://dist.apache.org/repos/dist\
>>                /dev/beam/<version>/python
>>
>> We wouldn't have to built a second artifact or change a version number.
>> However, I understand if we want the additional convenience of:
>>
>>    pip install --pre apache_beam
>>
>> As Robert mentioned, this would just be the additional step to rename
>> the version number and push the RC package to PyPi. Not necessary in my
>> opinion but I don't mind.
>>
>
> Value proposition is for downstream library providers. They can update
> their setup.py file to test beam rc in combination with rest of their other
> dependencies and test that combination with ease. I agree it is not
> necessary for us or any other party that just wants to test beam packages
> and its direct dependencies in isolation.
>
>
>>
>> Thanks,
>> Max
>>
>> On 02.05.19 19:43, Ahmet Altay wrote:
>> >
>> >
>> > On Thu, May 2, 2019 at 9:29 AM Robert Bradshaw <robertwb@google.com
>> > <ma...@google.com>> wrote:
>> >
>> >     On Thu, May 2, 2019 at 6:03 PM Michael Luckey <adude3141@gmail.com
>> >     <ma...@gmail.com>> wrote:
>> >      >
>> >      > Yes, I understood this. But I m personally more paranoid about
>> >     releasing.
>> >      >
>> >      > So formally vote (and corresponding testing) was done on rc. If
>> >     we rebuild and resign, wouldn't that mean we also need to revote?
>> >
>> >     Yeah, that's the sticking point. I suppose we could build the
>> packages
>> >     with rc tags, push them to pypi, and also build them without rc
>> tags,
>> >     and push those (and the full source tarball, which doesn't have an
>> rc
>> >     tag either) to svn, and have the vote officially cover what's in svn
>> >     but the rc ones are just for convenience.
>> >
>> >
>> > I prefer this way of double building. I think modifying the release
>> > process as in your initial proposal would be risky. In that proposal,
>> > what we test for validations would be different than what we push out
>> as
>> > final releases. The modification I see would be:
>> >
>> > 2c) Get the artifacts from svn, un-package, change the version to
>> > include the rc number, and re-package and push. (This is very similar
>> to
>> > the process we use for building the wheel files today, with the
>> > exception of changing the version part.)
>> >
>> >     (But, given that I can "pip
>> >     install https::svn.apache.org/path/to/tarball
>> >     <http://svn.apache.org/path/to/tarball>" it'd primarily have
>> >     value for others doing "pip install --pre".)
>> >
>> >
>> >     This is regardless of whether is OK per apache to publish such
>> binary
>> >     blobs to a third party place (though IMHO it follows the intent of
>> the
>> >     release process).
>> >
>> >
>> > This is also my understanding. What is being proposed here aligns with
>> > what Kenn found out in the other threads.
>> >
>> >
>> >      > If I understand correctly, there will be some changed version
>> >     string in distributed sources (setup.py?). So there is some binary
>> >     difference. And just talking about me, doing that repackaging I
>> >     would certainly mess it up and package some unwanted changes.
>> >
>> >     We definitely would not want this to be a manual step--I wouldn't
>> >     trust myself :).
>> >
>> >
>> >      > On Thu, May 2, 2019 at 5:43 PM Robert Bradshaw
>> >     <robertwb@google.com <ma...@google.com>> wrote:
>> >      >>
>> >      >> On Thu, May 2, 2019 at 5:24 PM Michael Luckey
>> >     <adude3141@gmail.com <ma...@gmail.com>> wrote:
>> >      >> >
>> >      >> > Thanks Ahmet for calling out to the airflow folks. I believe,
>> >     I am able to follow their argument. So from my point of view I do
>> >     not have an issue with apache policy. But honestly still trying to
>> >     wrap my head around Roberts concern with rebuilding/resigning.
>> >     Currently, our actual release is only a tag on source repo and
>> >     promoting artefacts. Do not yet understand how that needs to change
>> >     to get PyPi included.
>> >      >>
>> >      >> It's not a big change, but let me clarify.
>> >      >>
>> >      >> Currently our release preparation goes something like this:
>> >      >>
>> >      >> 1) Check out the repo, update the versions to 2.x, build and
>> >     sign the artifacts.
>> >      >> 2) Announce these artifacts as rcN
>> >      >> 2a) Push the artifacts to SVN dev/...
>> >      >> 2b) Push artifacts to the apache maven repository.
>> >      >> 3) Depending on vote, go back to step (1) or forward to step
>> (4).
>> >      >> 4) Copy these artifacts as the actual release.
>> >      >>
>> >      >> Now if we just try to add (2c) Push these artifacts to Pypi, it
>> will
>> >      >> be treated (by pypi's tooling, anyone who downloads the tarball,
>> >     ...)
>> >      >> as an actual release. You also can't re-push a tarball with the
>> same
>> >      >> name and different contents (the idea being that named releases
>> >     should
>> >      >> never change). So we'd need to change step (1) to update the
>> version
>> >      >> to 2.x.rcN *and* add a step in (4) to update the version to 2.x
>> >     (no rc
>> >      >> suffix), rebuild, resign before publishing.
>> >      >>
>> >      >> As mentioned, possibly the rcN suffix could be part of the
>> building
>> >      >> step for Python.
>> >      >>
>> >      >> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <altay@google.com
>> >     <ma...@google.com>> wrote:
>> >      >> >>
>> >      >> >> Michael, Max and other folks who are concerned about the
>> >     compatibility with the apache release policy. Does the information
>> >     in this thread sufficiently address your concerns? Especially the
>> >     part where, the rc artifacts will be protected by a flag (i.e.
>> >     --pre) from general consumption.
>> >      >> >>
>> >      >> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw
>> >     <robertwb@google.com <ma...@google.com>> wrote:
>> >      >> >>>
>> >      >> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay
>> >     <altay@google.com <ma...@google.com>> wrote:
>> >      >> >>> >
>> >      >> >>> > This conversation get quite Python centric. Is there a
>> >     similar need for Java?
>> >      >> >>>
>> >      >> >>> I think Java is already covered. Go is a different story
>> >     (but the even
>> >      >> >>> versioning and releasing is being worked out).
>> >      >> >>>
>> >      >> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw
>> >     <robertwb@google.com <ma...@google.com>> wrote:
>> >      >> >>> >>
>> >      >> >>> >> If we can, by the apache guidelines, post RCs to pypy
>> that is
>> >      >> >>> >> definitely the way to go. (Note that test.pypi is for
>> >     developing
>> >      >> >>> >> against the pypi interface, not for pushing anything
>> >     real.) The caveat
>> >      >> >>> >> about naming these with rcN in the version number still
>> >     applies
>> >      >> >>> >> (that's how pypi guards them against non-explicit
>> installs).
>> >      >> >>> >
>> >      >> >>> > Related to the caveat, I believe this can be easily
>> >     scripted or even made part of the travis/wheels pipeline to take the
>> >     release branch, edit the version string in place to add rc, and
>> >     build the necessary files.
>> >      >> >>>
>> >      >> >>> Yes. But the resulting artifacts would have to be rebuilt
>> (and
>> >      >> >>> re-signed) without the version edit for the actual release.
>> >     (Well, we
>> >      >> >>> could possibly edit the artifacts rather than rebuild
>> them.) And
>> >      >> >>> pushing un-edited ones early would be really bad. (It's the
>> >     classic
>> >      >> >>> tension of whether a pre-release should be marked
>> internally or
>> >      >> >>> externally, re-publishing a new set of bits for the actual
>> >     release or
>> >      >> >>> re-using version numbers for different sets of bits. Pypi
>> >     does one,
>> >      >> >>> apache does another...)
>> >      >> >>>
>> >      >> >>> >> The advantage is that a user can do "pip install --pre
>> >     apache-beam" to
>> >      >> >>> >> get the latest rc rather than "pip install
>> >      >> >>> >>
>> >
>> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"
>> >      >> >>> >>
>> >      >> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada
>> >     <pabloem@google.com <ma...@google.com>> wrote:
>> >      >> >>> >> >
>> >      >> >>> >> > Aw that's interesting!
>> >      >> >>> >> >
>> >      >> >>> >> > I think, with these considerations, I am only
>> >     marginally more inclined towards publishing to test.pypi. That would
>> >     make me a +0.9 on publishing RCs to the main pip repo then.
>> >      >> >>> >> >
>> >      >> >>> >> > Thanks for doing the research Ahmet. :)
>> >      >> >>> >> > Best
>> >      >> >>> >> > -P
>> >      >> >>> >> >
>> >      >> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay
>> >     <altay@google.com <ma...@google.com>> wrote:
>> >      >> >>> >> >>
>> >      >> >>> >> >> I asked to Airflow folks about this. See [1] for the
>> >     full response and a link to one of their RC emails. To summarize
>> >     their position (specifically for pypi) is: Unless a user does
>> >     something explicit (such as using a flag, or explicitly requesting
>> >     an rc release), pip install will not serve RC binaries. And that is
>> >     compatible with RC section of
>> >     http://www.apache.org/legal/release-policy.html#release-types
>> >      >> >>> >> >>
>> >      >> >>> >> >> Ahmet
>> >      >> >>> >> >>
>> >      >> >>> >> >> [1]
>> >
>> https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E
>> >      >> >>> >> >>
>> >      >> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay
>> >     <altay@google.com <ma...@google.com>> wrote:
>> >      >> >>> >> >>>
>> >      >> >>> >> >>> The incremental value of publishing python artifacts
>> >     to a separate place but not to actual pypi listing will be low.
>> >     Users can already download RC artifacts, or even pip install from
>> >     http location directly. I think the incremental value will be low,
>> >     because for a user or a downstream library to test with Beam RCs
>> >     using their usual ways will still require them to get other
>> >     dependencies from the regular pypi listing. That would mean they
>> >     need to change their setup to test with beam rcs, which is the same
>> >     state as today. There will be some incremental value of putting them
>> >     in more obvious places (e.g. pypi test repository). I would rather
>> >     not complicate the release process for doing this.
>> >      >> >>> >> >>>
>> >      >> >>> >> >>>
>> >      >> >>> >> >>>
>> >      >> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles
>> >     <kenn@apache.org <ma...@apache.org>> wrote:
>> >      >> >>> >> >>>>
>> >      >> >>> >> >>>> Pip is also able to be pointed at any raw hosted
>> >     directory for the install, right? So we could publish RCs or
>> >     snapshots somewhere with more obvious caveats and not interfere with
>> >     the pypi list of actual releases. Much like the Java snapshots are
>> >     stored in a separate opt-in repository.
>> >      >> >>> >> >>>>
>> >      >> >>> >> >>>> Kenn
>> >      >> >>> >> >>>>
>> >      >> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels
>> >     <mxm@apache.org <ma...@apache.org>> wrote:
>> >      >> >>> >> >>>>>
>> >      >> >>> >> >>>>> > wouldn't that be in conflict with Apache release
>> >     policy [1] ?
>> >      >> >>> >> >>>>> > [1]
>> http://www.apache.org/legal/release-policy.html
>> >      >> >>> >> >>>>>
>> >      >> >>> >> >>>>> Indeed, advertising pre-release artifacts is
>> >     against ASF rules. For
>> >      >> >>> >> >>>>> example, Flink was asked to remove a link to the
>> >     Maven snapshot
>> >      >> >>> >> >>>>> repository from their download page.
>> >      >> >>> >> >>>>>
>> >      >> >>> >> >>>>> However, that does not mean we cannot publish
>> >     Python artifacts. We just
>> >      >> >>> >> >>>>> have to clearly mark them for developers only and
>> >     not advertise them
>> >      >> >>> >> >>>>> alongside with the official releases.
>> >      >> >>> >> >>>>>
>> >      >> >>> >> >>>>> -Max
>> >      >> >>> >> >>>>>
>> >      >> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote:
>> >      >> >>> >> >>>>> > Don't we push java artifacts to maven
>> >     repositories as part of the RC
>> >      >> >>> >> >>>>> > process? And completely unvetted snapshots? (Or
>> >     is this OK because
>> >      >> >>> >> >>>>> > they are special opt-in apache-only ones?)
>> >      >> >>> >> >>>>> >
>> >      >> >>> >> >>>>> > I am generally in favor of the idea, but would
>> >     like to avoid increased
>> >      >> >>> >> >>>>> > toil on the release manager.
>> >      >> >>> >> >>>>> >
>> >      >> >>> >> >>>>> > One potential hitch I see is that current release
>> >     process updates the
>> >      >> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release
>> >     indicator in the version
>> >      >> >>> >> >>>>> > number) whereas pypi (and other systems)
>> >     typically expect distinct
>> >      >> >>> >> >>>>> > (recognizable) version numbers for each attempt,
>> >     and only the actual
>> >      >> >>> >> >>>>> > final result has the actual final release
>> version.
>> >      >> >>> >> >>>>> >
>> >      >> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay
>> >     <altay@google.com <ma...@google.com>> wrote:
>> >      >> >>> >> >>>>> >>
>> >      >> >>> >> >>>>> >> I do not know the answer.I believe this will be
>> >     similar to sharing the RC artifacts for validation purposes and
>> >     would not be a formal release by itself. But I am not an expert and
>> >     I hope others will share their opinions.
>> >      >> >>> >> >>>>> >>
>> >      >> >>> >> >>>>> >> I quickly searched pypi for apache projects and
>> >     found at least airflow [1] and libcloud [2] are publishing rc
>> >     artifacts to pypi. We can reach out to those communities and learn
>> >     about their processes.
>> >      >> >>> >> >>>>> >>
>> >      >> >>> >> >>>>> >> Ahmet
>> >      >> >>> >> >>>>> >>
>> >      >> >>> >> >>>>> >> [1]
>> https://pypi.org/project/apache-airflow/#history
>> >      >> >>> >> >>>>> >> [2]
>> >     https://pypi.org/project/apache-libcloud/#history
>> >      >> >>> >> >>>>> >>
>> >      >> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey
>> >     <adude3141@gmail.com <ma...@gmail.com>> wrote:
>> >      >> >>> >> >>>>> >>>
>> >      >> >>> >> >>>>> >>> Hi,
>> >      >> >>> >> >>>>> >>>
>> >      >> >>> >> >>>>> >>> wouldn't that be in conflict with Apache
>> >     release policy [1] ?
>> >      >> >>> >> >>>>> >>>
>> >      >> >>> >> >>>>> >>> [1]
>> http://www.apache.org/legal/release-policy.html
>> >      >> >>> >> >>>>> >>>
>> >      >> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold
>> >     <amyrvold@google.com <ma...@google.com>> wrote:
>> >      >> >>> >> >>>>> >>>>
>> >      >> >>> >> >>>>> >>>> Great idea. I like the RC candidates to follow
>> >     as much as the release artifact process as possible.
>> >      >> >>> >> >>>>> >>>>
>> >      >> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay
>> >     <altay@google.com <ma...@google.com>> wrote:
>> >      >> >>> >> >>>>> >>>>>
>> >      >> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing
>> >     publishing to the production pypi repository with an rc tag in the
>> >     version. And in turn allow users to depend on beam's rc version +
>> >     all the other regular dependencies users would have directly from
>> pypi.
>> >      >> >>> >> >>>>> >>>>>
>> >      >> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be
>> >     helpful if test pypi repo also mirrors other packages that exist in
>> >     the production pypi repository.
>> >      >> >>> >> >>>>> >>>>>
>> >      >> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada
>> >     <pabloem@google.com <ma...@google.com>> wrote:
>> >      >> >>> >> >>>>> >>>>>>
>> >      >> >>> >> >>>>> >>>>>> I think this is a great idea. A way of doing
>> >     it for python would be by using the test repository for PyPi[1], and
>> >     that way we would not have to do an official PyPi release, but still
>> >     would be able to install it with pip (by passing an extra flag), and
>> >     test.
>> >      >> >>> >> >>>>> >>>>>>
>> >      >> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts
>> >     already in there[2]. At some point I looked into this, but couldn't
>> >     figure out who has access/the password for it.
>> >      >> >>> >> >>>>> >>>>>
>> >      >> >>> >> >>>>> >>>>>
>> >      >> >>> >> >>>>> >>>>> I also don't know who owns beam package in
>> >     test pypi repo. Does anybody know?
>> >      >> >>> >> >>>>> >>>>>
>> >      >> >>> >> >>>>> >>>>>>
>> >      >> >>> >> >>>>> >>>>>>
>> >      >> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the
>> >     test PyPi repo to avoid publishing to the main PyPi repo.
>> >      >> >>> >> >>>>> >>>>>> Best
>> >      >> >>> >> >>>>> >>>>>> -P.
>> >      >> >>> >> >>>>> >>>>>>
>> >      >> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/
>> >      >> >>> >> >>>>> >>>>>> [2]
>> https://test.pypi.org/project/apache-beam/
>> >      >> >>> >> >>>>> >>>>>>
>> >      >> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay
>> >     <altay@google.com <ma...@google.com>> wrote:
>> >      >> >>> >> >>>>> >>>>>>>
>> >      >> >>> >> >>>>> >>>>>>> Hi all,
>> >      >> >>> >> >>>>> >>>>>>>
>> >      >> >>> >> >>>>> >>>>>>> What do you think about the idea of
>> >     publishing pre-release artifacts as part of the RC emails?
>> >      >> >>> >> >>>>> >>>>>>>
>> >      >> >>> >> >>>>> >>>>>>> For Python this would translate into
>> >     publishing the same artifacts from RC email with a version like
>> >     "2.X.0rcY" to pypi. I do not know, but I am guessing we can do a
>> >     similar thing with Maven central for Java artifacts as well.
>> >      >> >>> >> >>>>> >>>>>>>
>> >      >> >>> >> >>>>> >>>>>>> Advantages would be:
>> >      >> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their
>> >     own purposes using the same exact process they will normally use.
>> >      >> >>> >> >>>>> >>>>>>>   - Enable early-adaptors to start using RC
>> >     releases early on in the release cycle if that is what they would
>> >     like to do. This will in turn reduce time pressure on some releases.
>> >     Especially for cases like someone needs a release to be finalized
>> >     for an upcoming event.
>> >      >> >>> >> >>>>> >>>>>>>
>> >      >> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I
>> >     could think of:
>> >      >> >>> >> >>>>> >>>>>>> - Users could request support for RC
>> >     artifacts. Hopefully in the form of feedback for us to improve the
>> >     release. But it could also be in the form of folks using RC
>> >     artifacts for production for a long time.
>> >      >> >>> >> >>>>> >>>>>>> - It will add toil to the current release
>> >     process, there will be one more step for each RC. I think for python
>> >     this will be a small step but nevertheless it will be additional
>> work.
>> >      >> >>> >> >>>>> >>>>>>>
>> >      >> >>> >> >>>>> >>>>>>> For an example of this, you can take a look
>> >     at tensorflow releases. For 1.13 there were 3 pre-releases [1].
>> >      >> >>> >> >>>>> >>>>>>>
>> >      >> >>> >> >>>>> >>>>>>> Ahmet
>> >      >> >>> >> >>>>> >>>>>>>
>> >      >> >>> >> >>>>> >>>>>>> [1]
>> >     https://pypi.org/project/tensorflow/#history
>> >
>>
>

Re: [Discuss] Publishing pre-release artifacts to repositories

[Ahmet Altay <al...@google.com>]

Thank you Max. Michael, does the current state of the proposal address your
concerns?

*From: *Maximilian Michels <mx...@apache.org>
*Date: *Mon, May 6, 2019 at 8:32 AM
*To: * <de...@beam.apache.org>

Thanks for the link to the distribution guideline thread. The proposed
> process sounds good to me.
>
> Concerning Michael's point, I think we should follow the same procedure
> as we do for releasing to Maven Central, i.e. create a staging
> repository for validation which then gets released.
>
> The easiest way to do that, is to directly use the artifacts that we
> create during the release process:
>
>    pip install https://dist.apache.org/repos/dist\
>                /dev/beam/<version>/python
>
> We wouldn't have to built a second artifact or change a version number.
> However, I understand if we want the additional convenience of:
>
>    pip install --pre apache_beam
>
> As Robert mentioned, this would just be the additional step to rename
> the version number and push the RC package to PyPi. Not necessary in my
> opinion but I don't mind.
>

Value proposition is for downstream library providers. They can update
their setup.py file to test beam rc in combination with rest of their other
dependencies and test that combination with ease. I agree it is not
necessary for us or any other party that just wants to test beam packages
and its direct dependencies in isolation.


>
> Thanks,
> Max
>
> On 02.05.19 19:43, Ahmet Altay wrote:
> >
> >
> > On Thu, May 2, 2019 at 9:29 AM Robert Bradshaw <robertwb@google.com
> > <ma...@google.com>> wrote:
> >
> >     On Thu, May 2, 2019 at 6:03 PM Michael Luckey <adude3141@gmail.com
> >     <ma...@gmail.com>> wrote:
> >      >
> >      > Yes, I understood this. But I m personally more paranoid about
> >     releasing.
> >      >
> >      > So formally vote (and corresponding testing) was done on rc. If
> >     we rebuild and resign, wouldn't that mean we also need to revote?
> >
> >     Yeah, that's the sticking point. I suppose we could build the
> packages
> >     with rc tags, push them to pypi, and also build them without rc tags,
> >     and push those (and the full source tarball, which doesn't have an rc
> >     tag either) to svn, and have the vote officially cover what's in svn
> >     but the rc ones are just for convenience.
> >
> >
> > I prefer this way of double building. I think modifying the release
> > process as in your initial proposal would be risky. In that proposal,
> > what we test for validations would be different than what we push out as
> > final releases. The modification I see would be:
> >
> > 2c) Get the artifacts from svn, un-package, change the version to
> > include the rc number, and re-package and push. (This is very similar to
> > the process we use for building the wheel files today, with the
> > exception of changing the version part.)
> >
> >     (But, given that I can "pip
> >     install https::svn.apache.org/path/to/tarball
> >     <http://svn.apache.org/path/to/tarball>" it'd primarily have
> >     value for others doing "pip install --pre".)
> >
> >
> >     This is regardless of whether is OK per apache to publish such binary
> >     blobs to a third party place (though IMHO it follows the intent of
> the
> >     release process).
> >
> >
> > This is also my understanding. What is being proposed here aligns with
> > what Kenn found out in the other threads.
> >
> >
> >      > If I understand correctly, there will be some changed version
> >     string in distributed sources (setup.py?). So there is some binary
> >     difference. And just talking about me, doing that repackaging I
> >     would certainly mess it up and package some unwanted changes.
> >
> >     We definitely would not want this to be a manual step--I wouldn't
> >     trust myself :).
> >
> >
> >      > On Thu, May 2, 2019 at 5:43 PM Robert Bradshaw
> >     <robertwb@google.com <ma...@google.com>> wrote:
> >      >>
> >      >> On Thu, May 2, 2019 at 5:24 PM Michael Luckey
> >     <adude3141@gmail.com <ma...@gmail.com>> wrote:
> >      >> >
> >      >> > Thanks Ahmet for calling out to the airflow folks. I believe,
> >     I am able to follow their argument. So from my point of view I do
> >     not have an issue with apache policy. But honestly still trying to
> >     wrap my head around Roberts concern with rebuilding/resigning.
> >     Currently, our actual release is only a tag on source repo and
> >     promoting artefacts. Do not yet understand how that needs to change
> >     to get PyPi included.
> >      >>
> >      >> It's not a big change, but let me clarify.
> >      >>
> >      >> Currently our release preparation goes something like this:
> >      >>
> >      >> 1) Check out the repo, update the versions to 2.x, build and
> >     sign the artifacts.
> >      >> 2) Announce these artifacts as rcN
> >      >> 2a) Push the artifacts to SVN dev/...
> >      >> 2b) Push artifacts to the apache maven repository.
> >      >> 3) Depending on vote, go back to step (1) or forward to step (4).
> >      >> 4) Copy these artifacts as the actual release.
> >      >>
> >      >> Now if we just try to add (2c) Push these artifacts to Pypi, it
> will
> >      >> be treated (by pypi's tooling, anyone who downloads the tarball,
> >     ...)
> >      >> as an actual release. You also can't re-push a tarball with the
> same
> >      >> name and different contents (the idea being that named releases
> >     should
> >      >> never change). So we'd need to change step (1) to update the
> version
> >      >> to 2.x.rcN *and* add a step in (4) to update the version to 2.x
> >     (no rc
> >      >> suffix), rebuild, resign before publishing.
> >      >>
> >      >> As mentioned, possibly the rcN suffix could be part of the
> building
> >      >> step for Python.
> >      >>
> >      >> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <altay@google.com
> >     <ma...@google.com>> wrote:
> >      >> >>
> >      >> >> Michael, Max and other folks who are concerned about the
> >     compatibility with the apache release policy. Does the information
> >     in this thread sufficiently address your concerns? Especially the
> >     part where, the rc artifacts will be protected by a flag (i.e.
> >     --pre) from general consumption.
> >      >> >>
> >      >> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw
> >     <robertwb@google.com <ma...@google.com>> wrote:
> >      >> >>>
> >      >> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay
> >     <altay@google.com <ma...@google.com>> wrote:
> >      >> >>> >
> >      >> >>> > This conversation get quite Python centric. Is there a
> >     similar need for Java?
> >      >> >>>
> >      >> >>> I think Java is already covered. Go is a different story
> >     (but the even
> >      >> >>> versioning and releasing is being worked out).
> >      >> >>>
> >      >> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw
> >     <robertwb@google.com <ma...@google.com>> wrote:
> >      >> >>> >>
> >      >> >>> >> If we can, by the apache guidelines, post RCs to pypy
> that is
> >      >> >>> >> definitely the way to go. (Note that test.pypi is for
> >     developing
> >      >> >>> >> against the pypi interface, not for pushing anything
> >     real.) The caveat
> >      >> >>> >> about naming these with rcN in the version number still
> >     applies
> >      >> >>> >> (that's how pypi guards them against non-explicit
> installs).
> >      >> >>> >
> >      >> >>> > Related to the caveat, I believe this can be easily
> >     scripted or even made part of the travis/wheels pipeline to take the
> >     release branch, edit the version string in place to add rc, and
> >     build the necessary files.
> >      >> >>>
> >      >> >>> Yes. But the resulting artifacts would have to be rebuilt
> (and
> >      >> >>> re-signed) without the version edit for the actual release.
> >     (Well, we
> >      >> >>> could possibly edit the artifacts rather than rebuild them.)
> And
> >      >> >>> pushing un-edited ones early would be really bad. (It's the
> >     classic
> >      >> >>> tension of whether a pre-release should be marked internally
> or
> >      >> >>> externally, re-publishing a new set of bits for the actual
> >     release or
> >      >> >>> re-using version numbers for different sets of bits. Pypi
> >     does one,
> >      >> >>> apache does another...)
> >      >> >>>
> >      >> >>> >> The advantage is that a user can do "pip install --pre
> >     apache-beam" to
> >      >> >>> >> get the latest rc rather than "pip install
> >      >> >>> >>
> >
> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"
> >      >> >>> >>
> >      >> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada
> >     <pabloem@google.com <ma...@google.com>> wrote:
> >      >> >>> >> >
> >      >> >>> >> > Aw that's interesting!
> >      >> >>> >> >
> >      >> >>> >> > I think, with these considerations, I am only
> >     marginally more inclined towards publishing to test.pypi. That would
> >     make me a +0.9 on publishing RCs to the main pip repo then.
> >      >> >>> >> >
> >      >> >>> >> > Thanks for doing the research Ahmet. :)
> >      >> >>> >> > Best
> >      >> >>> >> > -P
> >      >> >>> >> >
> >      >> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay
> >     <altay@google.com <ma...@google.com>> wrote:
> >      >> >>> >> >>
> >      >> >>> >> >> I asked to Airflow folks about this. See [1] for the
> >     full response and a link to one of their RC emails. To summarize
> >     their position (specifically for pypi) is: Unless a user does
> >     something explicit (such as using a flag, or explicitly requesting
> >     an rc release), pip install will not serve RC binaries. And that is
> >     compatible with RC section of
> >     http://www.apache.org/legal/release-policy.html#release-types
> >      >> >>> >> >>
> >      >> >>> >> >> Ahmet
> >      >> >>> >> >>
> >      >> >>> >> >> [1]
> >
> https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E
> >      >> >>> >> >>
> >      >> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay
> >     <altay@google.com <ma...@google.com>> wrote:
> >      >> >>> >> >>>
> >      >> >>> >> >>> The incremental value of publishing python artifacts
> >     to a separate place but not to actual pypi listing will be low.
> >     Users can already download RC artifacts, or even pip install from
> >     http location directly. I think the incremental value will be low,
> >     because for a user or a downstream library to test with Beam RCs
> >     using their usual ways will still require them to get other
> >     dependencies from the regular pypi listing. That would mean they
> >     need to change their setup to test with beam rcs, which is the same
> >     state as today. There will be some incremental value of putting them
> >     in more obvious places (e.g. pypi test repository). I would rather
> >     not complicate the release process for doing this.
> >      >> >>> >> >>>
> >      >> >>> >> >>>
> >      >> >>> >> >>>
> >      >> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles
> >     <kenn@apache.org <ma...@apache.org>> wrote:
> >      >> >>> >> >>>>
> >      >> >>> >> >>>> Pip is also able to be pointed at any raw hosted
> >     directory for the install, right? So we could publish RCs or
> >     snapshots somewhere with more obvious caveats and not interfere with
> >     the pypi list of actual releases. Much like the Java snapshots are
> >     stored in a separate opt-in repository.
> >      >> >>> >> >>>>
> >      >> >>> >> >>>> Kenn
> >      >> >>> >> >>>>
> >      >> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels
> >     <mxm@apache.org <ma...@apache.org>> wrote:
> >      >> >>> >> >>>>>
> >      >> >>> >> >>>>> > wouldn't that be in conflict with Apache release
> >     policy [1] ?
> >      >> >>> >> >>>>> > [1]
> http://www.apache.org/legal/release-policy.html
> >      >> >>> >> >>>>>
> >      >> >>> >> >>>>> Indeed, advertising pre-release artifacts is
> >     against ASF rules. For
> >      >> >>> >> >>>>> example, Flink was asked to remove a link to the
> >     Maven snapshot
> >      >> >>> >> >>>>> repository from their download page.
> >      >> >>> >> >>>>>
> >      >> >>> >> >>>>> However, that does not mean we cannot publish
> >     Python artifacts. We just
> >      >> >>> >> >>>>> have to clearly mark them for developers only and
> >     not advertise them
> >      >> >>> >> >>>>> alongside with the official releases.
> >      >> >>> >> >>>>>
> >      >> >>> >> >>>>> -Max
> >      >> >>> >> >>>>>
> >      >> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote:
> >      >> >>> >> >>>>> > Don't we push java artifacts to maven
> >     repositories as part of the RC
> >      >> >>> >> >>>>> > process? And completely unvetted snapshots? (Or
> >     is this OK because
> >      >> >>> >> >>>>> > they are special opt-in apache-only ones?)
> >      >> >>> >> >>>>> >
> >      >> >>> >> >>>>> > I am generally in favor of the idea, but would
> >     like to avoid increased
> >      >> >>> >> >>>>> > toil on the release manager.
> >      >> >>> >> >>>>> >
> >      >> >>> >> >>>>> > One potential hitch I see is that current release
> >     process updates the
> >      >> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release
> >     indicator in the version
> >      >> >>> >> >>>>> > number) whereas pypi (and other systems)
> >     typically expect distinct
> >      >> >>> >> >>>>> > (recognizable) version numbers for each attempt,
> >     and only the actual
> >      >> >>> >> >>>>> > final result has the actual final release version.
> >      >> >>> >> >>>>> >
> >      >> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay
> >     <altay@google.com <ma...@google.com>> wrote:
> >      >> >>> >> >>>>> >>
> >      >> >>> >> >>>>> >> I do not know the answer.I believe this will be
> >     similar to sharing the RC artifacts for validation purposes and
> >     would not be a formal release by itself. But I am not an expert and
> >     I hope others will share their opinions.
> >      >> >>> >> >>>>> >>
> >      >> >>> >> >>>>> >> I quickly searched pypi for apache projects and
> >     found at least airflow [1] and libcloud [2] are publishing rc
> >     artifacts to pypi. We can reach out to those communities and learn
> >     about their processes.
> >      >> >>> >> >>>>> >>
> >      >> >>> >> >>>>> >> Ahmet
> >      >> >>> >> >>>>> >>
> >      >> >>> >> >>>>> >> [1]
> https://pypi.org/project/apache-airflow/#history
> >      >> >>> >> >>>>> >> [2]
> >     https://pypi.org/project/apache-libcloud/#history
> >      >> >>> >> >>>>> >>
> >      >> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey
> >     <adude3141@gmail.com <ma...@gmail.com>> wrote:
> >      >> >>> >> >>>>> >>>
> >      >> >>> >> >>>>> >>> Hi,
> >      >> >>> >> >>>>> >>>
> >      >> >>> >> >>>>> >>> wouldn't that be in conflict with Apache
> >     release policy [1] ?
> >      >> >>> >> >>>>> >>>
> >      >> >>> >> >>>>> >>> [1]
> http://www.apache.org/legal/release-policy.html
> >      >> >>> >> >>>>> >>>
> >      >> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold
> >     <amyrvold@google.com <ma...@google.com>> wrote:
> >      >> >>> >> >>>>> >>>>
> >      >> >>> >> >>>>> >>>> Great idea. I like the RC candidates to follow
> >     as much as the release artifact process as possible.
> >      >> >>> >> >>>>> >>>>
> >      >> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay
> >     <altay@google.com <ma...@google.com>> wrote:
> >      >> >>> >> >>>>> >>>>>
> >      >> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing
> >     publishing to the production pypi repository with an rc tag in the
> >     version. And in turn allow users to depend on beam's rc version +
> >     all the other regular dependencies users would have directly from
> pypi.
> >      >> >>> >> >>>>> >>>>>
> >      >> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be
> >     helpful if test pypi repo also mirrors other packages that exist in
> >     the production pypi repository.
> >      >> >>> >> >>>>> >>>>>
> >      >> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada
> >     <pabloem@google.com <ma...@google.com>> wrote:
> >      >> >>> >> >>>>> >>>>>>
> >      >> >>> >> >>>>> >>>>>> I think this is a great idea. A way of doing
> >     it for python would be by using the test repository for PyPi[1], and
> >     that way we would not have to do an official PyPi release, but still
> >     would be able to install it with pip (by passing an extra flag), and
> >     test.
> >      >> >>> >> >>>>> >>>>>>
> >      >> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts
> >     already in there[2]. At some point I looked into this, but couldn't
> >     figure out who has access/the password for it.
> >      >> >>> >> >>>>> >>>>>
> >      >> >>> >> >>>>> >>>>>
> >      >> >>> >> >>>>> >>>>> I also don't know who owns beam package in
> >     test pypi repo. Does anybody know?
> >      >> >>> >> >>>>> >>>>>
> >      >> >>> >> >>>>> >>>>>>
> >      >> >>> >> >>>>> >>>>>>
> >      >> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the
> >     test PyPi repo to avoid publishing to the main PyPi repo.
> >      >> >>> >> >>>>> >>>>>> Best
> >      >> >>> >> >>>>> >>>>>> -P.
> >      >> >>> >> >>>>> >>>>>>
> >      >> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/
> >      >> >>> >> >>>>> >>>>>> [2]
> https://test.pypi.org/project/apache-beam/
> >      >> >>> >> >>>>> >>>>>>
> >      >> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay
> >     <altay@google.com <ma...@google.com>> wrote:
> >      >> >>> >> >>>>> >>>>>>>
> >      >> >>> >> >>>>> >>>>>>> Hi all,
> >      >> >>> >> >>>>> >>>>>>>
> >      >> >>> >> >>>>> >>>>>>> What do you think about the idea of
> >     publishing pre-release artifacts as part of the RC emails?
> >      >> >>> >> >>>>> >>>>>>>
> >      >> >>> >> >>>>> >>>>>>> For Python this would translate into
> >     publishing the same artifacts from RC email with a version like
> >     "2.X.0rcY" to pypi. I do not know, but I am guessing we can do a
> >     similar thing with Maven central for Java artifacts as well.
> >      >> >>> >> >>>>> >>>>>>>
> >      >> >>> >> >>>>> >>>>>>> Advantages would be:
> >      >> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their
> >     own purposes using the same exact process they will normally use.
> >      >> >>> >> >>>>> >>>>>>>   - Enable early-adaptors to start using RC
> >     releases early on in the release cycle if that is what they would
> >     like to do. This will in turn reduce time pressure on some releases.
> >     Especially for cases like someone needs a release to be finalized
> >     for an upcoming event.
> >      >> >>> >> >>>>> >>>>>>>
> >      >> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I
> >     could think of:
> >      >> >>> >> >>>>> >>>>>>> - Users could request support for RC
> >     artifacts. Hopefully in the form of feedback for us to improve the
> >     release. But it could also be in the form of folks using RC
> >     artifacts for production for a long time.
> >      >> >>> >> >>>>> >>>>>>> - It will add toil to the current release
> >     process, there will be one more step for each RC. I think for python
> >     this will be a small step but nevertheless it will be additional
> work.
> >      >> >>> >> >>>>> >>>>>>>
> >      >> >>> >> >>>>> >>>>>>> For an example of this, you can take a look
> >     at tensorflow releases. For 1.13 there were 3 pre-releases [1].
> >      >> >>> >> >>>>> >>>>>>>
> >      >> >>> >> >>>>> >>>>>>> Ahmet
> >      >> >>> >> >>>>> >>>>>>>
> >      >> >>> >> >>>>> >>>>>>> [1]
> >     https://pypi.org/project/tensorflow/#history
> >
>

Re: [Discuss] Publishing pre-release artifacts to repositories

[Maximilian Michels <mx...@apache.org>]

Thanks for the link to the distribution guideline thread. The proposed 
process sounds good to me.

Concerning Michael's point, I think we should follow the same procedure 
as we do for releasing to Maven Central, i.e. create a staging 
repository for validation which then gets released.

The easiest way to do that, is to directly use the artifacts that we 
create during the release process:

   pip install https://dist.apache.org/repos/dist\
               /dev/beam/<version>/python

We wouldn't have to built a second artifact or change a version number. 
However, I understand if we want the additional convenience of:

   pip install --pre apache_beam

As Robert mentioned, this would just be the additional step to rename 
the version number and push the RC package to PyPi. Not necessary in my 
opinion but I don't mind.

Thanks,
Max

On 02.05.19 19:43, Ahmet Altay wrote:
> 
> 
> On Thu, May 2, 2019 at 9:29 AM Robert Bradshaw <robertwb@google.com 
> <ma...@google.com>> wrote:
> 
>     On Thu, May 2, 2019 at 6:03 PM Michael Luckey <adude3141@gmail.com
>     <ma...@gmail.com>> wrote:
>      >
>      > Yes, I understood this. But I m personally more paranoid about
>     releasing.
>      >
>      > So formally vote (and corresponding testing) was done on rc. If
>     we rebuild and resign, wouldn't that mean we also need to revote?
> 
>     Yeah, that's the sticking point. I suppose we could build the packages
>     with rc tags, push them to pypi, and also build them without rc tags,
>     and push those (and the full source tarball, which doesn't have an rc
>     tag either) to svn, and have the vote officially cover what's in svn
>     but the rc ones are just for convenience. 
> 
> 
> I prefer this way of double building. I think modifying the release 
> process as in your initial proposal would be risky. In that proposal, 
> what we test for validations would be different than what we push out as 
> final releases. The modification I see would be:
> 
> 2c) Get the artifacts from svn, un-package, change the version to 
> include the rc number, and re-package and push. (This is very similar to 
> the process we use for building the wheel files today, with the 
> exception of changing the version part.)
> 
>     (But, given that I can "pip
>     install https::svn.apache.org/path/to/tarball
>     <http://svn.apache.org/path/to/tarball>" it'd primarily have
>     value for others doing "pip install --pre".) 
> 
> 
>     This is regardless of whether is OK per apache to publish such binary
>     blobs to a third party place (though IMHO it follows the intent of the
>     release process).
> 
> 
> This is also my understanding. What is being proposed here aligns with 
> what Kenn found out in the other threads.
> 
> 
>      > If I understand correctly, there will be some changed version
>     string in distributed sources (setup.py?). So there is some binary
>     difference. And just talking about me, doing that repackaging I
>     would certainly mess it up and package some unwanted changes.
> 
>     We definitely would not want this to be a manual step--I wouldn't
>     trust myself :). 
> 
> 
>      > On Thu, May 2, 2019 at 5:43 PM Robert Bradshaw
>     <robertwb@google.com <ma...@google.com>> wrote:
>      >>
>      >> On Thu, May 2, 2019 at 5:24 PM Michael Luckey
>     <adude3141@gmail.com <ma...@gmail.com>> wrote:
>      >> >
>      >> > Thanks Ahmet for calling out to the airflow folks. I believe,
>     I am able to follow their argument. So from my point of view I do
>     not have an issue with apache policy. But honestly still trying to
>     wrap my head around Roberts concern with rebuilding/resigning.
>     Currently, our actual release is only a tag on source repo and
>     promoting artefacts. Do not yet understand how that needs to change
>     to get PyPi included.
>      >>
>      >> It's not a big change, but let me clarify.
>      >>
>      >> Currently our release preparation goes something like this:
>      >>
>      >> 1) Check out the repo, update the versions to 2.x, build and
>     sign the artifacts.
>      >> 2) Announce these artifacts as rcN
>      >> 2a) Push the artifacts to SVN dev/...
>      >> 2b) Push artifacts to the apache maven repository.
>      >> 3) Depending on vote, go back to step (1) or forward to step (4).
>      >> 4) Copy these artifacts as the actual release.
>      >>
>      >> Now if we just try to add (2c) Push these artifacts to Pypi, it will
>      >> be treated (by pypi's tooling, anyone who downloads the tarball,
>     ...)
>      >> as an actual release. You also can't re-push a tarball with the same
>      >> name and different contents (the idea being that named releases
>     should
>      >> never change). So we'd need to change step (1) to update the version
>      >> to 2.x.rcN *and* add a step in (4) to update the version to 2.x
>     (no rc
>      >> suffix), rebuild, resign before publishing.
>      >>
>      >> As mentioned, possibly the rcN suffix could be part of the building
>      >> step for Python.
>      >>
>      >> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <altay@google.com
>     <ma...@google.com>> wrote:
>      >> >>
>      >> >> Michael, Max and other folks who are concerned about the
>     compatibility with the apache release policy. Does the information
>     in this thread sufficiently address your concerns? Especially the
>     part where, the rc artifacts will be protected by a flag (i.e.
>     --pre) from general consumption.
>      >> >>
>      >> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw
>     <robertwb@google.com <ma...@google.com>> wrote:
>      >> >>>
>      >> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay
>     <altay@google.com <ma...@google.com>> wrote:
>      >> >>> >
>      >> >>> > This conversation get quite Python centric. Is there a
>     similar need for Java?
>      >> >>>
>      >> >>> I think Java is already covered. Go is a different story
>     (but the even
>      >> >>> versioning and releasing is being worked out).
>      >> >>>
>      >> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw
>     <robertwb@google.com <ma...@google.com>> wrote:
>      >> >>> >>
>      >> >>> >> If we can, by the apache guidelines, post RCs to pypy that is
>      >> >>> >> definitely the way to go. (Note that test.pypi is for
>     developing
>      >> >>> >> against the pypi interface, not for pushing anything
>     real.) The caveat
>      >> >>> >> about naming these with rcN in the version number still
>     applies
>      >> >>> >> (that's how pypi guards them against non-explicit installs).
>      >> >>> >
>      >> >>> > Related to the caveat, I believe this can be easily
>     scripted or even made part of the travis/wheels pipeline to take the
>     release branch, edit the version string in place to add rc, and
>     build the necessary files.
>      >> >>>
>      >> >>> Yes. But the resulting artifacts would have to be rebuilt (and
>      >> >>> re-signed) without the version edit for the actual release.
>     (Well, we
>      >> >>> could possibly edit the artifacts rather than rebuild them.) And
>      >> >>> pushing un-edited ones early would be really bad. (It's the
>     classic
>      >> >>> tension of whether a pre-release should be marked internally or
>      >> >>> externally, re-publishing a new set of bits for the actual
>     release or
>      >> >>> re-using version numbers for different sets of bits. Pypi
>     does one,
>      >> >>> apache does another...)
>      >> >>>
>      >> >>> >> The advantage is that a user can do "pip install --pre
>     apache-beam" to
>      >> >>> >> get the latest rc rather than "pip install
>      >> >>> >>
>     https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"
>      >> >>> >>
>      >> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada
>     <pabloem@google.com <ma...@google.com>> wrote:
>      >> >>> >> >
>      >> >>> >> > Aw that's interesting!
>      >> >>> >> >
>      >> >>> >> > I think, with these considerations, I am only
>     marginally more inclined towards publishing to test.pypi. That would
>     make me a +0.9 on publishing RCs to the main pip repo then.
>      >> >>> >> >
>      >> >>> >> > Thanks for doing the research Ahmet. :)
>      >> >>> >> > Best
>      >> >>> >> > -P
>      >> >>> >> >
>      >> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay
>     <altay@google.com <ma...@google.com>> wrote:
>      >> >>> >> >>
>      >> >>> >> >> I asked to Airflow folks about this. See [1] for the
>     full response and a link to one of their RC emails. To summarize
>     their position (specifically for pypi) is: Unless a user does
>     something explicit (such as using a flag, or explicitly requesting
>     an rc release), pip install will not serve RC binaries. And that is
>     compatible with RC section of
>     http://www.apache.org/legal/release-policy.html#release-types
>      >> >>> >> >>
>      >> >>> >> >> Ahmet
>      >> >>> >> >>
>      >> >>> >> >> [1]
>     https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E
>      >> >>> >> >>
>      >> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay
>     <altay@google.com <ma...@google.com>> wrote:
>      >> >>> >> >>>
>      >> >>> >> >>> The incremental value of publishing python artifacts
>     to a separate place but not to actual pypi listing will be low.
>     Users can already download RC artifacts, or even pip install from
>     http location directly. I think the incremental value will be low,
>     because for a user or a downstream library to test with Beam RCs
>     using their usual ways will still require them to get other
>     dependencies from the regular pypi listing. That would mean they
>     need to change their setup to test with beam rcs, which is the same
>     state as today. There will be some incremental value of putting them
>     in more obvious places (e.g. pypi test repository). I would rather
>     not complicate the release process for doing this.
>      >> >>> >> >>>
>      >> >>> >> >>>
>      >> >>> >> >>>
>      >> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles
>     <kenn@apache.org <ma...@apache.org>> wrote:
>      >> >>> >> >>>>
>      >> >>> >> >>>> Pip is also able to be pointed at any raw hosted
>     directory for the install, right? So we could publish RCs or
>     snapshots somewhere with more obvious caveats and not interfere with
>     the pypi list of actual releases. Much like the Java snapshots are
>     stored in a separate opt-in repository.
>      >> >>> >> >>>>
>      >> >>> >> >>>> Kenn
>      >> >>> >> >>>>
>      >> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels
>     <mxm@apache.org <ma...@apache.org>> wrote:
>      >> >>> >> >>>>>
>      >> >>> >> >>>>> > wouldn't that be in conflict with Apache release
>     policy [1] ?
>      >> >>> >> >>>>> > [1] http://www.apache.org/legal/release-policy.html
>      >> >>> >> >>>>>
>      >> >>> >> >>>>> Indeed, advertising pre-release artifacts is
>     against ASF rules. For
>      >> >>> >> >>>>> example, Flink was asked to remove a link to the
>     Maven snapshot
>      >> >>> >> >>>>> repository from their download page.
>      >> >>> >> >>>>>
>      >> >>> >> >>>>> However, that does not mean we cannot publish
>     Python artifacts. We just
>      >> >>> >> >>>>> have to clearly mark them for developers only and
>     not advertise them
>      >> >>> >> >>>>> alongside with the official releases.
>      >> >>> >> >>>>>
>      >> >>> >> >>>>> -Max
>      >> >>> >> >>>>>
>      >> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote:
>      >> >>> >> >>>>> > Don't we push java artifacts to maven
>     repositories as part of the RC
>      >> >>> >> >>>>> > process? And completely unvetted snapshots? (Or
>     is this OK because
>      >> >>> >> >>>>> > they are special opt-in apache-only ones?)
>      >> >>> >> >>>>> >
>      >> >>> >> >>>>> > I am generally in favor of the idea, but would
>     like to avoid increased
>      >> >>> >> >>>>> > toil on the release manager.
>      >> >>> >> >>>>> >
>      >> >>> >> >>>>> > One potential hitch I see is that current release
>     process updates the
>      >> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release
>     indicator in the version
>      >> >>> >> >>>>> > number) whereas pypi (and other systems)
>     typically expect distinct
>      >> >>> >> >>>>> > (recognizable) version numbers for each attempt,
>     and only the actual
>      >> >>> >> >>>>> > final result has the actual final release version.
>      >> >>> >> >>>>> >
>      >> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay
>     <altay@google.com <ma...@google.com>> wrote:
>      >> >>> >> >>>>> >>
>      >> >>> >> >>>>> >> I do not know the answer.I believe this will be
>     similar to sharing the RC artifacts for validation purposes and
>     would not be a formal release by itself. But I am not an expert and
>     I hope others will share their opinions.
>      >> >>> >> >>>>> >>
>      >> >>> >> >>>>> >> I quickly searched pypi for apache projects and
>     found at least airflow [1] and libcloud [2] are publishing rc
>     artifacts to pypi. We can reach out to those communities and learn
>     about their processes.
>      >> >>> >> >>>>> >>
>      >> >>> >> >>>>> >> Ahmet
>      >> >>> >> >>>>> >>
>      >> >>> >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history
>      >> >>> >> >>>>> >> [2]
>     https://pypi.org/project/apache-libcloud/#history
>      >> >>> >> >>>>> >>
>      >> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey
>     <adude3141@gmail.com <ma...@gmail.com>> wrote:
>      >> >>> >> >>>>> >>>
>      >> >>> >> >>>>> >>> Hi,
>      >> >>> >> >>>>> >>>
>      >> >>> >> >>>>> >>> wouldn't that be in conflict with Apache
>     release policy [1] ?
>      >> >>> >> >>>>> >>>
>      >> >>> >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html
>      >> >>> >> >>>>> >>>
>      >> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold
>     <amyrvold@google.com <ma...@google.com>> wrote:
>      >> >>> >> >>>>> >>>>
>      >> >>> >> >>>>> >>>> Great idea. I like the RC candidates to follow
>     as much as the release artifact process as possible.
>      >> >>> >> >>>>> >>>>
>      >> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay
>     <altay@google.com <ma...@google.com>> wrote:
>      >> >>> >> >>>>> >>>>>
>      >> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing
>     publishing to the production pypi repository with an rc tag in the
>     version. And in turn allow users to depend on beam's rc version +
>     all the other regular dependencies users would have directly from pypi.
>      >> >>> >> >>>>> >>>>>
>      >> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be
>     helpful if test pypi repo also mirrors other packages that exist in
>     the production pypi repository.
>      >> >>> >> >>>>> >>>>>
>      >> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada
>     <pabloem@google.com <ma...@google.com>> wrote:
>      >> >>> >> >>>>> >>>>>>
>      >> >>> >> >>>>> >>>>>> I think this is a great idea. A way of doing
>     it for python would be by using the test repository for PyPi[1], and
>     that way we would not have to do an official PyPi release, but still
>     would be able to install it with pip (by passing an extra flag), and
>     test.
>      >> >>> >> >>>>> >>>>>>
>      >> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts
>     already in there[2]. At some point I looked into this, but couldn't
>     figure out who has access/the password for it.
>      >> >>> >> >>>>> >>>>>
>      >> >>> >> >>>>> >>>>>
>      >> >>> >> >>>>> >>>>> I also don't know who owns beam package in
>     test pypi repo. Does anybody know?
>      >> >>> >> >>>>> >>>>>
>      >> >>> >> >>>>> >>>>>>
>      >> >>> >> >>>>> >>>>>>
>      >> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the
>     test PyPi repo to avoid publishing to the main PyPi repo.
>      >> >>> >> >>>>> >>>>>> Best
>      >> >>> >> >>>>> >>>>>> -P.
>      >> >>> >> >>>>> >>>>>>
>      >> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/
>      >> >>> >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/
>      >> >>> >> >>>>> >>>>>>
>      >> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay
>     <altay@google.com <ma...@google.com>> wrote:
>      >> >>> >> >>>>> >>>>>>>
>      >> >>> >> >>>>> >>>>>>> Hi all,
>      >> >>> >> >>>>> >>>>>>>
>      >> >>> >> >>>>> >>>>>>> What do you think about the idea of
>     publishing pre-release artifacts as part of the RC emails?
>      >> >>> >> >>>>> >>>>>>>
>      >> >>> >> >>>>> >>>>>>> For Python this would translate into
>     publishing the same artifacts from RC email with a version like
>     "2.X.0rcY" to pypi. I do not know, but I am guessing we can do a
>     similar thing with Maven central for Java artifacts as well.
>      >> >>> >> >>>>> >>>>>>>
>      >> >>> >> >>>>> >>>>>>> Advantages would be:
>      >> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their
>     own purposes using the same exact process they will normally use.
>      >> >>> >> >>>>> >>>>>>>   - Enable early-adaptors to start using RC
>     releases early on in the release cycle if that is what they would
>     like to do. This will in turn reduce time pressure on some releases.
>     Especially for cases like someone needs a release to be finalized
>     for an upcoming event.
>      >> >>> >> >>>>> >>>>>>>
>      >> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I
>     could think of:
>      >> >>> >> >>>>> >>>>>>> - Users could request support for RC
>     artifacts. Hopefully in the form of feedback for us to improve the
>     release. But it could also be in the form of folks using RC
>     artifacts for production for a long time.
>      >> >>> >> >>>>> >>>>>>> - It will add toil to the current release
>     process, there will be one more step for each RC. I think for python
>     this will be a small step but nevertheless it will be additional work.
>      >> >>> >> >>>>> >>>>>>>
>      >> >>> >> >>>>> >>>>>>> For an example of this, you can take a look
>     at tensorflow releases. For 1.13 there were 3 pre-releases [1].
>      >> >>> >> >>>>> >>>>>>>
>      >> >>> >> >>>>> >>>>>>> Ahmet
>      >> >>> >> >>>>> >>>>>>>
>      >> >>> >> >>>>> >>>>>>> [1]
>     https://pypi.org/project/tensorflow/#history
> 

Re: [Discuss] Publishing pre-release artifacts to repositories

[Ahmet Altay <al...@google.com>]

On Thu, May 2, 2019 at 9:29 AM Robert Bradshaw <ro...@google.com> wrote:

> On Thu, May 2, 2019 at 6:03 PM Michael Luckey <ad...@gmail.com> wrote:
> >
> > Yes, I understood this. But I m personally more paranoid about releasing.
> >
> > So formally vote (and corresponding testing) was done on rc. If we
> rebuild and resign, wouldn't that mean we also need to revote?
>
> Yeah, that's the sticking point. I suppose we could build the packages
> with rc tags, push them to pypi, and also build them without rc tags,
> and push those (and the full source tarball, which doesn't have an rc
> tag either) to svn, and have the vote officially cover what's in svn
> but the rc ones are just for convenience.


I prefer this way of double building. I think modifying the release process
as in your initial proposal would be risky. In that proposal, what we test
for validations would be different than what we push out as final releases.
The modification I see would be:

2c) Get the artifacts from svn, un-package, change the version to include
the rc number, and re-package and push. (This is very similar to the
process we use for building the wheel files today, with the exception of
changing the version part.)


> (But, given that I can "pip
> install https::svn.apache.org/path/to/tarball" it'd primarily have
> value for others doing "pip install --pre".)


> This is regardless of whether is OK per apache to publish such binary
> blobs to a third party place (though IMHO it follows the intent of the
> release process).
>

This is also my understanding. What is being proposed here aligns with what
Kenn found out in the other threads.


>
> > If I understand correctly, there will be some changed version string in
> distributed sources (setup.py?). So there is some binary difference. And
> just talking about me, doing that repackaging I would certainly mess it up
> and package some unwanted changes.
>
> We definitely would not want this to be a manual step--I wouldn't
> trust myself :).


> > On Thu, May 2, 2019 at 5:43 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>
> >> On Thu, May 2, 2019 at 5:24 PM Michael Luckey <ad...@gmail.com>
> wrote:
> >> >
> >> > Thanks Ahmet for calling out to the airflow folks. I believe, I am
> able to follow their argument. So from my point of view I do not have an
> issue with apache policy. But honestly still trying to wrap my head around
> Roberts concern with rebuilding/resigning. Currently, our actual release is
> only a tag on source repo and promoting artefacts. Do not yet understand
> how that needs to change to get PyPi included.
> >>
> >> It's not a big change, but let me clarify.
> >>
> >> Currently our release preparation goes something like this:
> >>
> >> 1) Check out the repo, update the versions to 2.x, build and sign the
> artifacts.
> >> 2) Announce these artifacts as rcN
> >> 2a) Push the artifacts to SVN dev/...
> >> 2b) Push artifacts to the apache maven repository.
> >> 3) Depending on vote, go back to step (1) or forward to step (4).
> >> 4) Copy these artifacts as the actual release.
> >>
> >> Now if we just try to add (2c) Push these artifacts to Pypi, it will
> >> be treated (by pypi's tooling, anyone who downloads the tarball, ...)
> >> as an actual release. You also can't re-push a tarball with the same
> >> name and different contents (the idea being that named releases should
> >> never change). So we'd need to change step (1) to update the version
> >> to 2.x.rcN *and* add a step in (4) to update the version to 2.x (no rc
> >> suffix), rebuild, resign before publishing.
> >>
> >> As mentioned, possibly the rcN suffix could be part of the building
> >> step for Python.
> >>
> >> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <al...@google.com> wrote:
> >> >>
> >> >> Michael, Max and other folks who are concerned about the
> compatibility with the apache release policy. Does the information in this
> thread sufficiently address your concerns? Especially the part where, the
> rc artifacts will be protected by a flag (i.e. --pre) from general
> consumption.
> >> >>
> >> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >> >>>
> >> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay <al...@google.com>
> wrote:
> >> >>> >
> >> >>> > This conversation get quite Python centric. Is there a similar
> need for Java?
> >> >>>
> >> >>> I think Java is already covered. Go is a different story (but the
> even
> >> >>> versioning and releasing is being worked out).
> >> >>>
> >> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw <
> robertwb@google.com> wrote:
> >> >>> >>
> >> >>> >> If we can, by the apache guidelines, post RCs to pypy that is
> >> >>> >> definitely the way to go. (Note that test.pypi is for developing
> >> >>> >> against the pypi interface, not for pushing anything real.) The
> caveat
> >> >>> >> about naming these with rcN in the version number still applies
> >> >>> >> (that's how pypi guards them against non-explicit installs).
> >> >>> >
> >> >>> > Related to the caveat, I believe this can be easily scripted or
> even made part of the travis/wheels pipeline to take the release branch,
> edit the version string in place to add rc, and build the necessary files.
> >> >>>
> >> >>> Yes. But the resulting artifacts would have to be rebuilt (and
> >> >>> re-signed) without the version edit for the actual release. (Well,
> we
> >> >>> could possibly edit the artifacts rather than rebuild them.) And
> >> >>> pushing un-edited ones early would be really bad. (It's the classic
> >> >>> tension of whether a pre-release should be marked internally or
> >> >>> externally, re-publishing a new set of bits for the actual release
> or
> >> >>> re-using version numbers for different sets of bits. Pypi does one,
> >> >>> apache does another...)
> >> >>>
> >> >>> >> The advantage is that a user can do "pip install --pre
> apache-beam" to
> >> >>> >> get the latest rc rather than "pip install
> >> >>> >>
> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"
> >> >>> >>
> >> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada <
> pabloem@google.com> wrote:
> >> >>> >> >
> >> >>> >> > Aw that's interesting!
> >> >>> >> >
> >> >>> >> > I think, with these considerations, I am only marginally more
> inclined towards publishing to test.pypi. That would make me a +0.9 on
> publishing RCs to the main pip repo then.
> >> >>> >> >
> >> >>> >> > Thanks for doing the research Ahmet. :)
> >> >>> >> > Best
> >> >>> >> > -P
> >> >>> >> >
> >> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <al...@google.com>
> wrote:
> >> >>> >> >>
> >> >>> >> >> I asked to Airflow folks about this. See [1] for the full
> response and a link to one of their RC emails. To summarize their position
> (specifically for pypi) is: Unless a user does something explicit (such as
> using a flag, or explicitly requesting an rc release), pip install will not
> serve RC binaries. And that is compatible with RC section of
> http://www.apache.org/legal/release-policy.html#release-types
> >> >>> >> >>
> >> >>> >> >> Ahmet
> >> >>> >> >>
> >> >>> >> >> [1]
> https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E
> >> >>> >> >>
> >> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <al...@google.com>
> wrote:
> >> >>> >> >>>
> >> >>> >> >>> The incremental value of publishing python artifacts to a
> separate place but not to actual pypi listing will be low. Users can
> already download RC artifacts, or even pip install from http location
> directly. I think the incremental value will be low, because for a user or
> a downstream library to test with Beam RCs using their usual ways will
> still require them to get other dependencies from the regular pypi listing.
> That would mean they need to change their setup to test with beam rcs,
> which is the same state as today. There will be some incremental value of
> putting them in more obvious places (e.g. pypi test repository). I would
> rather not complicate the release process for doing this.
> >> >>> >> >>>
> >> >>> >> >>>
> >> >>> >> >>>
> >> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles <
> kenn@apache.org> wrote:
> >> >>> >> >>>>
> >> >>> >> >>>> Pip is also able to be pointed at any raw hosted directory
> for the install, right? So we could publish RCs or snapshots somewhere with
> more obvious caveats and not interfere with the pypi list of actual
> releases. Much like the Java snapshots are stored in a separate opt-in
> repository.
> >> >>> >> >>>>
> >> >>> >> >>>> Kenn
> >> >>> >> >>>>
> >> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels <
> mxm@apache.org> wrote:
> >> >>> >> >>>>>
> >> >>> >> >>>>> > wouldn't that be in conflict with Apache release policy
> [1] ?
> >> >>> >> >>>>> > [1] http://www.apache.org/legal/release-policy.html
> >> >>> >> >>>>>
> >> >>> >> >>>>> Indeed, advertising pre-release artifacts is against ASF
> rules. For
> >> >>> >> >>>>> example, Flink was asked to remove a link to the Maven
> snapshot
> >> >>> >> >>>>> repository from their download page.
> >> >>> >> >>>>>
> >> >>> >> >>>>> However, that does not mean we cannot publish Python
> artifacts. We just
> >> >>> >> >>>>> have to clearly mark them for developers only and not
> advertise them
> >> >>> >> >>>>> alongside with the official releases.
> >> >>> >> >>>>>
> >> >>> >> >>>>> -Max
> >> >>> >> >>>>>
> >> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote:
> >> >>> >> >>>>> > Don't we push java artifacts to maven repositories as
> part of the RC
> >> >>> >> >>>>> > process? And completely unvetted snapshots? (Or is this
> OK because
> >> >>> >> >>>>> > they are special opt-in apache-only ones?)
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > I am generally in favor of the idea, but would like to
> avoid increased
> >> >>> >> >>>>> > toil on the release manager.
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > One potential hitch I see is that current release
> process updates the
> >> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release indicator
> in the version
> >> >>> >> >>>>> > number) whereas pypi (and other systems) typically
> expect distinct
> >> >>> >> >>>>> > (recognizable) version numbers for each attempt, and
> only the actual
> >> >>> >> >>>>> > final result has the actual final release version.
> >> >>> >> >>>>> >
> >> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay <
> altay@google.com> wrote:
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> I do not know the answer.I believe this will be similar
> to sharing the RC artifacts for validation purposes and would not be a
> formal release by itself. But I am not an expert and I hope others will
> share their opinions.
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> I quickly searched pypi for apache projects and found
> at least airflow [1] and libcloud [2] are publishing rc artifacts to pypi.
> We can reach out to those communities and learn about their processes.
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> Ahmet
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history
> >> >>> >> >>>>> >> [2] https://pypi.org/project/apache-libcloud/#history
> >> >>> >> >>>>> >>
> >> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey <
> adude3141@gmail.com> wrote:
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> Hi,
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> wouldn't that be in conflict with Apache release
> policy [1] ?
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html
> >> >>> >> >>>>> >>>
> >> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold <
> amyrvold@google.com> wrote:
> >> >>> >> >>>>> >>>>
> >> >>> >> >>>>> >>>> Great idea. I like the RC candidates to follow as
> much as the release artifact process as possible.
> >> >>> >> >>>>> >>>>
> >> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay <
> altay@google.com> wrote:
> >> >>> >> >>>>> >>>>>
> >> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing publishing to
> the production pypi repository with an rc tag in the version. And in turn
> allow users to depend on beam's rc version + all the other regular
> dependencies users would have directly from pypi.
> >> >>> >> >>>>> >>>>>
> >> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be helpful
> if test pypi repo also mirrors other packages that exist in the production
> pypi repository.
> >> >>> >> >>>>> >>>>>
> >> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada <
> pabloem@google.com> wrote:
> >> >>> >> >>>>> >>>>>>
> >> >>> >> >>>>> >>>>>> I think this is a great idea. A way of doing it for
> python would be by using the test repository for PyPi[1], and that way we
> would not have to do an official PyPi release, but still would be able to
> install it with pip (by passing an extra flag), and test.
> >> >>> >> >>>>> >>>>>>
> >> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts already in
> there[2]. At some point I looked into this, but couldn't figure out who has
> access/the password for it.
> >> >>> >> >>>>> >>>>>
> >> >>> >> >>>>> >>>>>
> >> >>> >> >>>>> >>>>> I also don't know who owns beam package in test pypi
> repo. Does anybody know?
> >> >>> >> >>>>> >>>>>
> >> >>> >> >>>>> >>>>>>
> >> >>> >> >>>>> >>>>>>
> >> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the test
> PyPi repo to avoid publishing to the main PyPi repo.
> >> >>> >> >>>>> >>>>>> Best
> >> >>> >> >>>>> >>>>>> -P.
> >> >>> >> >>>>> >>>>>>
> >> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/
> >> >>> >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/
> >> >>> >> >>>>> >>>>>>
> >> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay <
> altay@google.com> wrote:
> >> >>> >> >>>>> >>>>>>>
> >> >>> >> >>>>> >>>>>>> Hi all,
> >> >>> >> >>>>> >>>>>>>
> >> >>> >> >>>>> >>>>>>> What do you think about the idea of publishing
> pre-release artifacts as part of the RC emails?
> >> >>> >> >>>>> >>>>>>>
> >> >>> >> >>>>> >>>>>>> For Python this would translate into publishing
> the same artifacts from RC email with a version like "2.X.0rcY" to pypi. I
> do not know, but I am guessing we can do a similar thing with Maven central
> for Java artifacts as well.
> >> >>> >> >>>>> >>>>>>>
> >> >>> >> >>>>> >>>>>>> Advantages would be:
> >> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their own
> purposes using the same exact process they will normally use.
> >> >>> >> >>>>> >>>>>>>   - Enable early-adaptors to start using RC
> releases early on in the release cycle if that is what they would like to
> do. This will in turn reduce time pressure on some releases. Especially for
> cases like someone needs a release to be finalized for an upcoming event.
> >> >>> >> >>>>> >>>>>>>
> >> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I could
> think of:
> >> >>> >> >>>>> >>>>>>> - Users could request support for RC artifacts.
> Hopefully in the form of feedback for us to improve the release. But it
> could also be in the form of folks using RC artifacts for production for a
> long time.
> >> >>> >> >>>>> >>>>>>> - It will add toil to the current release process,
> there will be one more step for each RC. I think for python this will be a
> small step but nevertheless it will be additional work.
> >> >>> >> >>>>> >>>>>>>
> >> >>> >> >>>>> >>>>>>> For an example of this, you can take a look at
> tensorflow releases. For 1.13 there were 3 pre-releases [1].
> >> >>> >> >>>>> >>>>>>>
> >> >>> >> >>>>> >>>>>>> Ahmet
> >> >>> >> >>>>> >>>>>>>
> >> >>> >> >>>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history
>

Re: [Discuss] Publishing pre-release artifacts to repositories

[Robert Bradshaw <ro...@google.com>]

On Thu, May 2, 2019 at 6:03 PM Michael Luckey <ad...@gmail.com> wrote:
>
> Yes, I understood this. But I m personally more paranoid about releasing.
>
> So formally vote (and corresponding testing) was done on rc. If we rebuild and resign, wouldn't that mean we also need to revote?

Yeah, that's the sticking point. I suppose we could build the packages
with rc tags, push them to pypi, and also build them without rc tags,
and push those (and the full source tarball, which doesn't have an rc
tag either) to svn, and have the vote officially cover what's in svn
but the rc ones are just for convenience. (But, given that I can "pip
install https::svn.apache.org/path/to/tarball" it'd primarily have
value for others doing "pip install --pre".)

This is regardless of whether is OK per apache to publish such binary
blobs to a third party place (though IMHO it follows the intent of the
release process).

> If I understand correctly, there will be some changed version string in distributed sources (setup.py?). So there is some binary difference. And just talking about me, doing that repackaging I would certainly mess it up and package some unwanted changes.

We definitely would not want this to be a manual step--I wouldn't
trust myself :).

> On Thu, May 2, 2019 at 5:43 PM Robert Bradshaw <ro...@google.com> wrote:
>>
>> On Thu, May 2, 2019 at 5:24 PM Michael Luckey <ad...@gmail.com> wrote:
>> >
>> > Thanks Ahmet for calling out to the airflow folks. I believe, I am able to follow their argument. So from my point of view I do not have an issue with apache policy. But honestly still trying to wrap my head around Roberts concern with rebuilding/resigning. Currently, our actual release is only a tag on source repo and promoting artefacts. Do not yet understand how that needs to change to get PyPi included.
>>
>> It's not a big change, but let me clarify.
>>
>> Currently our release preparation goes something like this:
>>
>> 1) Check out the repo, update the versions to 2.x, build and sign the artifacts.
>> 2) Announce these artifacts as rcN
>> 2a) Push the artifacts to SVN dev/...
>> 2b) Push artifacts to the apache maven repository.
>> 3) Depending on vote, go back to step (1) or forward to step (4).
>> 4) Copy these artifacts as the actual release.
>>
>> Now if we just try to add (2c) Push these artifacts to Pypi, it will
>> be treated (by pypi's tooling, anyone who downloads the tarball, ...)
>> as an actual release. You also can't re-push a tarball with the same
>> name and different contents (the idea being that named releases should
>> never change). So we'd need to change step (1) to update the version
>> to 2.x.rcN *and* add a step in (4) to update the version to 2.x (no rc
>> suffix), rebuild, resign before publishing.
>>
>> As mentioned, possibly the rcN suffix could be part of the building
>> step for Python.
>>
>> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <al...@google.com> wrote:
>> >>
>> >> Michael, Max and other folks who are concerned about the compatibility with the apache release policy. Does the information in this thread sufficiently address your concerns? Especially the part where, the rc artifacts will be protected by a flag (i.e. --pre) from general consumption.
>> >>
>> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw <ro...@google.com> wrote:
>> >>>
>> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay <al...@google.com> wrote:
>> >>> >
>> >>> > This conversation get quite Python centric. Is there a similar need for Java?
>> >>>
>> >>> I think Java is already covered. Go is a different story (but the even
>> >>> versioning and releasing is being worked out).
>> >>>
>> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw <ro...@google.com> wrote:
>> >>> >>
>> >>> >> If we can, by the apache guidelines, post RCs to pypy that is
>> >>> >> definitely the way to go. (Note that test.pypi is for developing
>> >>> >> against the pypi interface, not for pushing anything real.) The caveat
>> >>> >> about naming these with rcN in the version number still applies
>> >>> >> (that's how pypi guards them against non-explicit installs).
>> >>> >
>> >>> > Related to the caveat, I believe this can be easily scripted or even made part of the travis/wheels pipeline to take the release branch, edit the version string in place to add rc, and build the necessary files.
>> >>>
>> >>> Yes. But the resulting artifacts would have to be rebuilt (and
>> >>> re-signed) without the version edit for the actual release. (Well, we
>> >>> could possibly edit the artifacts rather than rebuild them.) And
>> >>> pushing un-edited ones early would be really bad. (It's the classic
>> >>> tension of whether a pre-release should be marked internally or
>> >>> externally, re-publishing a new set of bits for the actual release or
>> >>> re-using version numbers for different sets of bits. Pypi does one,
>> >>> apache does another...)
>> >>>
>> >>> >> The advantage is that a user can do "pip install --pre apache-beam" to
>> >>> >> get the latest rc rather than "pip install
>> >>> >> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"
>> >>> >>
>> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada <pa...@google.com> wrote:
>> >>> >> >
>> >>> >> > Aw that's interesting!
>> >>> >> >
>> >>> >> > I think, with these considerations, I am only marginally more inclined towards publishing to test.pypi. That would make me a +0.9 on publishing RCs to the main pip repo then.
>> >>> >> >
>> >>> >> > Thanks for doing the research Ahmet. :)
>> >>> >> > Best
>> >>> >> > -P
>> >>> >> >
>> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <al...@google.com> wrote:
>> >>> >> >>
>> >>> >> >> I asked to Airflow folks about this. See [1] for the full response and a link to one of their RC emails. To summarize their position (specifically for pypi) is: Unless a user does something explicit (such as using a flag, or explicitly requesting an rc release), pip install will not serve RC binaries. And that is compatible with RC section of http://www.apache.org/legal/release-policy.html#release-types
>> >>> >> >>
>> >>> >> >> Ahmet
>> >>> >> >>
>> >>> >> >> [1] https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E
>> >>> >> >>
>> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <al...@google.com> wrote:
>> >>> >> >>>
>> >>> >> >>> The incremental value of publishing python artifacts to a separate place but not to actual pypi listing will be low. Users can already download RC artifacts, or even pip install from http location directly. I think the incremental value will be low, because for a user or a downstream library to test with Beam RCs using their usual ways will still require them to get other dependencies from the regular pypi listing. That would mean they need to change their setup to test with beam rcs, which is the same state as today. There will be some incremental value of putting them in more obvious places (e.g. pypi test repository). I would rather not complicate the release process for doing this.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles <ke...@apache.org> wrote:
>> >>> >> >>>>
>> >>> >> >>>> Pip is also able to be pointed at any raw hosted directory for the install, right? So we could publish RCs or snapshots somewhere with more obvious caveats and not interfere with the pypi list of actual releases. Much like the Java snapshots are stored in a separate opt-in repository.
>> >>> >> >>>>
>> >>> >> >>>> Kenn
>> >>> >> >>>>
>> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels <mx...@apache.org> wrote:
>> >>> >> >>>>>
>> >>> >> >>>>> > wouldn't that be in conflict with Apache release policy [1] ?
>> >>> >> >>>>> > [1] http://www.apache.org/legal/release-policy.html
>> >>> >> >>>>>
>> >>> >> >>>>> Indeed, advertising pre-release artifacts is against ASF rules. For
>> >>> >> >>>>> example, Flink was asked to remove a link to the Maven snapshot
>> >>> >> >>>>> repository from their download page.
>> >>> >> >>>>>
>> >>> >> >>>>> However, that does not mean we cannot publish Python artifacts. We just
>> >>> >> >>>>> have to clearly mark them for developers only and not advertise them
>> >>> >> >>>>> alongside with the official releases.
>> >>> >> >>>>>
>> >>> >> >>>>> -Max
>> >>> >> >>>>>
>> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote:
>> >>> >> >>>>> > Don't we push java artifacts to maven repositories as part of the RC
>> >>> >> >>>>> > process? And completely unvetted snapshots? (Or is this OK because
>> >>> >> >>>>> > they are special opt-in apache-only ones?)
>> >>> >> >>>>> >
>> >>> >> >>>>> > I am generally in favor of the idea, but would like to avoid increased
>> >>> >> >>>>> > toil on the release manager.
>> >>> >> >>>>> >
>> >>> >> >>>>> > One potential hitch I see is that current release process updates the
>> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release indicator in the version
>> >>> >> >>>>> > number) whereas pypi (and other systems) typically expect distinct
>> >>> >> >>>>> > (recognizable) version numbers for each attempt, and only the actual
>> >>> >> >>>>> > final result has the actual final release version.
>> >>> >> >>>>> >
>> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay <al...@google.com> wrote:
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> I do not know the answer.I believe this will be similar to sharing the RC artifacts for validation purposes and would not be a formal release by itself. But I am not an expert and I hope others will share their opinions.
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> I quickly searched pypi for apache projects and found at least airflow [1] and libcloud [2] are publishing rc artifacts to pypi. We can reach out to those communities and learn about their processes.
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> Ahmet
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history
>> >>> >> >>>>> >> [2] https://pypi.org/project/apache-libcloud/#history
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey <ad...@gmail.com> wrote:
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> Hi,
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> wouldn't that be in conflict with Apache release policy [1] ?
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold <am...@google.com> wrote:
>> >>> >> >>>>> >>>>
>> >>> >> >>>>> >>>> Great idea. I like the RC candidates to follow as much as the release artifact process as possible.
>> >>> >> >>>>> >>>>
>> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay <al...@google.com> wrote:
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing publishing to the production pypi repository with an rc tag in the version. And in turn allow users to depend on beam's rc version + all the other regular dependencies users would have directly from pypi.
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be helpful if test pypi repo also mirrors other packages that exist in the production pypi repository.
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada <pa...@google.com> wrote:
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> I think this is a great idea. A way of doing it for python would be by using the test repository for PyPi[1], and that way we would not have to do an official PyPi release, but still would be able to install it with pip (by passing an extra flag), and test.
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts already in there[2]. At some point I looked into this, but couldn't figure out who has access/the password for it.
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>> I also don't know who owns beam package in test pypi repo. Does anybody know?
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the test PyPi repo to avoid publishing to the main PyPi repo.
>> >>> >> >>>>> >>>>>> Best
>> >>> >> >>>>> >>>>>> -P.
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/
>> >>> >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay <al...@google.com> wrote:
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> Hi all,
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> What do you think about the idea of publishing pre-release artifacts as part of the RC emails?
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> For Python this would translate into publishing the same artifacts from RC email with a version like "2.X.0rcY" to pypi. I do not know, but I am guessing we can do a similar thing with Maven central for Java artifacts as well.
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> Advantages would be:
>> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their own purposes using the same exact process they will normally use.
>> >>> >> >>>>> >>>>>>>   - Enable early-adaptors to start using RC releases early on in the release cycle if that is what they would like to do. This will in turn reduce time pressure on some releases. Especially for cases like someone needs a release to be finalized for an upcoming event.
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I could think of:
>> >>> >> >>>>> >>>>>>> - Users could request support for RC artifacts. Hopefully in the form of feedback for us to improve the release. But it could also be in the form of folks using RC artifacts for production for a long time.
>> >>> >> >>>>> >>>>>>> - It will add toil to the current release process, there will be one more step for each RC. I think for python this will be a small step but nevertheless it will be additional work.
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> For an example of this, you can take a look at tensorflow releases. For 1.13 there were 3 pre-releases [1].
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> Ahmet
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history

Re: [Discuss] Publishing pre-release artifacts to repositories

[Michael Luckey <ad...@gmail.com>]

Yes, I understood this. But I m personally more paranoid about releasing.

So formally vote (and corresponding testing) was done on rc. If we rebuild
and resign, wouldn't that mean we also need to revote?

If I understand correctly, there will be some changed version string in
distributed sources (setup.py?). So there is some binary difference. And
just talking about me, doing that repackaging I would certainly mess it up
and package some unwanted changes.

On Thu, May 2, 2019 at 5:43 PM Robert Bradshaw <ro...@google.com> wrote:

> On Thu, May 2, 2019 at 5:24 PM Michael Luckey <ad...@gmail.com> wrote:
> >
> > Thanks Ahmet for calling out to the airflow folks. I believe, I am able
> to follow their argument. So from my point of view I do not have an issue
> with apache policy. But honestly still trying to wrap my head around
> Roberts concern with rebuilding/resigning. Currently, our actual release is
> only a tag on source repo and promoting artefacts. Do not yet understand
> how that needs to change to get PyPi included.
>
> It's not a big change, but let me clarify.
>
> Currently our release preparation goes something like this:
>
> 1) Check out the repo, update the versions to 2.x, build and sign the
> artifacts.
> 2) Announce these artifacts as rcN
> 2a) Push the artifacts to SVN dev/...
> 2b) Push artifacts to the apache maven repository.
> 3) Depending on vote, go back to step (1) or forward to step (4).
> 4) Copy these artifacts as the actual release.
>
> Now if we just try to add (2c) Push these artifacts to Pypi, it will
> be treated (by pypi's tooling, anyone who downloads the tarball, ...)
> as an actual release. You also can't re-push a tarball with the same
> name and different contents (the idea being that named releases should
> never change). So we'd need to change step (1) to update the version
> to 2.x.rcN *and* add a step in (4) to update the version to 2.x (no rc
> suffix), rebuild, resign before publishing.
>
> As mentioned, possibly the rcN suffix could be part of the building
> step for Python.
>
> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <al...@google.com> wrote:
> >>
> >> Michael, Max and other folks who are concerned about the compatibility
> with the apache release policy. Does the information in this thread
> sufficiently address your concerns? Especially the part where, the rc
> artifacts will be protected by a flag (i.e. --pre) from general consumption.
> >>
> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>>
> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay <al...@google.com> wrote:
> >>> >
> >>> > This conversation get quite Python centric. Is there a similar need
> for Java?
> >>>
> >>> I think Java is already covered. Go is a different story (but the even
> >>> versioning and releasing is being worked out).
> >>>
> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw <ro...@google.com>
> wrote:
> >>> >>
> >>> >> If we can, by the apache guidelines, post RCs to pypy that is
> >>> >> definitely the way to go. (Note that test.pypi is for developing
> >>> >> against the pypi interface, not for pushing anything real.) The
> caveat
> >>> >> about naming these with rcN in the version number still applies
> >>> >> (that's how pypi guards them against non-explicit installs).
> >>> >
> >>> > Related to the caveat, I believe this can be easily scripted or even
> made part of the travis/wheels pipeline to take the release branch, edit
> the version string in place to add rc, and build the necessary files.
> >>>
> >>> Yes. But the resulting artifacts would have to be rebuilt (and
> >>> re-signed) without the version edit for the actual release. (Well, we
> >>> could possibly edit the artifacts rather than rebuild them.) And
> >>> pushing un-edited ones early would be really bad. (It's the classic
> >>> tension of whether a pre-release should be marked internally or
> >>> externally, re-publishing a new set of bits for the actual release or
> >>> re-using version numbers for different sets of bits. Pypi does one,
> >>> apache does another...)
> >>>
> >>> >> The advantage is that a user can do "pip install --pre apache-beam"
> to
> >>> >> get the latest rc rather than "pip install
> >>> >>
> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"
> >>> >>
> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada <pa...@google.com>
> wrote:
> >>> >> >
> >>> >> > Aw that's interesting!
> >>> >> >
> >>> >> > I think, with these considerations, I am only marginally more
> inclined towards publishing to test.pypi. That would make me a +0.9 on
> publishing RCs to the main pip repo then.
> >>> >> >
> >>> >> > Thanks for doing the research Ahmet. :)
> >>> >> > Best
> >>> >> > -P
> >>> >> >
> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <al...@google.com>
> wrote:
> >>> >> >>
> >>> >> >> I asked to Airflow folks about this. See [1] for the full
> response and a link to one of their RC emails. To summarize their position
> (specifically for pypi) is: Unless a user does something explicit (such as
> using a flag, or explicitly requesting an rc release), pip install will not
> serve RC binaries. And that is compatible with RC section of
> http://www.apache.org/legal/release-policy.html#release-types
> >>> >> >>
> >>> >> >> Ahmet
> >>> >> >>
> >>> >> >> [1]
> https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E
> >>> >> >>
> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <al...@google.com>
> wrote:
> >>> >> >>>
> >>> >> >>> The incremental value of publishing python artifacts to a
> separate place but not to actual pypi listing will be low. Users can
> already download RC artifacts, or even pip install from http location
> directly. I think the incremental value will be low, because for a user or
> a downstream library to test with Beam RCs using their usual ways will
> still require them to get other dependencies from the regular pypi listing.
> That would mean they need to change their setup to test with beam rcs,
> which is the same state as today. There will be some incremental value of
> putting them in more obvious places (e.g. pypi test repository). I would
> rather not complicate the release process for doing this.
> >>> >> >>>
> >>> >> >>>
> >>> >> >>>
> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles <
> kenn@apache.org> wrote:
> >>> >> >>>>
> >>> >> >>>> Pip is also able to be pointed at any raw hosted directory for
> the install, right? So we could publish RCs or snapshots somewhere with
> more obvious caveats and not interfere with the pypi list of actual
> releases. Much like the Java snapshots are stored in a separate opt-in
> repository.
> >>> >> >>>>
> >>> >> >>>> Kenn
> >>> >> >>>>
> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels <
> mxm@apache.org> wrote:
> >>> >> >>>>>
> >>> >> >>>>> > wouldn't that be in conflict with Apache release policy [1]
> ?
> >>> >> >>>>> > [1] http://www.apache.org/legal/release-policy.html
> >>> >> >>>>>
> >>> >> >>>>> Indeed, advertising pre-release artifacts is against ASF
> rules. For
> >>> >> >>>>> example, Flink was asked to remove a link to the Maven
> snapshot
> >>> >> >>>>> repository from their download page.
> >>> >> >>>>>
> >>> >> >>>>> However, that does not mean we cannot publish Python
> artifacts. We just
> >>> >> >>>>> have to clearly mark them for developers only and not
> advertise them
> >>> >> >>>>> alongside with the official releases.
> >>> >> >>>>>
> >>> >> >>>>> -Max
> >>> >> >>>>>
> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote:
> >>> >> >>>>> > Don't we push java artifacts to maven repositories as part
> of the RC
> >>> >> >>>>> > process? And completely unvetted snapshots? (Or is this OK
> because
> >>> >> >>>>> > they are special opt-in apache-only ones?)
> >>> >> >>>>> >
> >>> >> >>>>> > I am generally in favor of the idea, but would like to
> avoid increased
> >>> >> >>>>> > toil on the release manager.
> >>> >> >>>>> >
> >>> >> >>>>> > One potential hitch I see is that current release process
> updates the
> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release indicator in
> the version
> >>> >> >>>>> > number) whereas pypi (and other systems) typically expect
> distinct
> >>> >> >>>>> > (recognizable) version numbers for each attempt, and only
> the actual
> >>> >> >>>>> > final result has the actual final release version.
> >>> >> >>>>> >
> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay <
> altay@google.com> wrote:
> >>> >> >>>>> >>
> >>> >> >>>>> >> I do not know the answer.I believe this will be similar to
> sharing the RC artifacts for validation purposes and would not be a formal
> release by itself. But I am not an expert and I hope others will share
> their opinions.
> >>> >> >>>>> >>
> >>> >> >>>>> >> I quickly searched pypi for apache projects and found at
> least airflow [1] and libcloud [2] are publishing rc artifacts to pypi. We
> can reach out to those communities and learn about their processes.
> >>> >> >>>>> >>
> >>> >> >>>>> >> Ahmet
> >>> >> >>>>> >>
> >>> >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history
> >>> >> >>>>> >> [2] https://pypi.org/project/apache-libcloud/#history
> >>> >> >>>>> >>
> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey <
> adude3141@gmail.com> wrote:
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> Hi,
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> wouldn't that be in conflict with Apache release policy
> [1] ?
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold <
> amyrvold@google.com> wrote:
> >>> >> >>>>> >>>>
> >>> >> >>>>> >>>> Great idea. I like the RC candidates to follow as much
> as the release artifact process as possible.
> >>> >> >>>>> >>>>
> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay <
> altay@google.com> wrote:
> >>> >> >>>>> >>>>>
> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing publishing to
> the production pypi repository with an rc tag in the version. And in turn
> allow users to depend on beam's rc version + all the other regular
> dependencies users would have directly from pypi.
> >>> >> >>>>> >>>>>
> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be helpful if
> test pypi repo also mirrors other packages that exist in the production
> pypi repository.
> >>> >> >>>>> >>>>>
> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada <
> pabloem@google.com> wrote:
> >>> >> >>>>> >>>>>>
> >>> >> >>>>> >>>>>> I think this is a great idea. A way of doing it for
> python would be by using the test repository for PyPi[1], and that way we
> would not have to do an official PyPi release, but still would be able to
> install it with pip (by passing an extra flag), and test.
> >>> >> >>>>> >>>>>>
> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts already in
> there[2]. At some point I looked into this, but couldn't figure out who has
> access/the password for it.
> >>> >> >>>>> >>>>>
> >>> >> >>>>> >>>>>
> >>> >> >>>>> >>>>> I also don't know who owns beam package in test pypi
> repo. Does anybody know?
> >>> >> >>>>> >>>>>
> >>> >> >>>>> >>>>>>
> >>> >> >>>>> >>>>>>
> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the test PyPi
> repo to avoid publishing to the main PyPi repo.
> >>> >> >>>>> >>>>>> Best
> >>> >> >>>>> >>>>>> -P.
> >>> >> >>>>> >>>>>>
> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/
> >>> >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/
> >>> >> >>>>> >>>>>>
> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay <
> altay@google.com> wrote:
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> Hi all,
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> What do you think about the idea of publishing
> pre-release artifacts as part of the RC emails?
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> For Python this would translate into publishing the
> same artifacts from RC email with a version like "2.X.0rcY" to pypi. I do
> not know, but I am guessing we can do a similar thing with Maven central
> for Java artifacts as well.
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> Advantages would be:
> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their own
> purposes using the same exact process they will normally use.
> >>> >> >>>>> >>>>>>>   - Enable early-adaptors to start using RC releases
> early on in the release cycle if that is what they would like to do. This
> will in turn reduce time pressure on some releases. Especially for cases
> like someone needs a release to be finalized for an upcoming event.
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I could think
> of:
> >>> >> >>>>> >>>>>>> - Users could request support for RC artifacts.
> Hopefully in the form of feedback for us to improve the release. But it
> could also be in the form of folks using RC artifacts for production for a
> long time.
> >>> >> >>>>> >>>>>>> - It will add toil to the current release process,
> there will be one more step for each RC. I think for python this will be a
> small step but nevertheless it will be additional work.
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> For an example of this, you can take a look at
> tensorflow releases. For 1.13 there were 3 pre-releases [1].
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> Ahmet
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history
>

Re: [Discuss] Publishing pre-release artifacts to repositories

[Ahmet Altay <al...@google.com>]

On Thu, May 2, 2019 at 8:56 AM Kenneth Knowles <ke...@apache.org> wrote:

> Pulling out the relevant pypi bit w.r.t. RCs:
>
>
>> - Release candidates, nightly or snapshots need to be clearly tagged as
>> pre-release on https://pypi.org/project/apache<project>/#history
>> - The latest version should not point to an artefact containing
>> unapproved code e.g. to a release candidate or snapshot
>>
>
> These guidelines were pending assessment by legal & infra. I don't know if
> there has been an update. It has been a few months.
>

Thanks a lot for finding all these information. IMO, this proposal fits
with the above guideline. Where can we track/ask about the information you
found specifically about pypi?


>
> Kenn
>
> On Thu, May 2, 2019 at 8:51 AM Kenneth Knowles <ke...@apache.org> wrote:
>
>> Ah, and here's one on general@incubator specifically about RCs:
>> https://lists.apache.org/thread.html/c4afcf0807d71f844d912a7e5fe6b481f0779bdcf88ccf9abe50a160@%3Cgeneral.incubator.apache.org%3E
>>
>> Kenn
>>
>> On Thu, May 2, 2019 at 8:49 AM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> I'd suggest looking for experience beyond Beam and Airflow. I don't see
>>> links to some relevant threads.
>>>
>>> Here's one from legal-discuss@ about binary channels and how they
>>> relate to source releases:
>>> https://lists.apache.org/thread.html/d578819f1afa6b8fb697ea72083e0fb05e43938a23d6e7bb804069b8@%3Clegal-discuss.apache.org%3E
>>>
>>> Here's one from the incubator about how RCs and tags relate to ASF
>>> release policy:
>>> https://lists.apache.org/thread.html/982077ef279e50b260d302d96685e40be6fcabdb0bd43d519621cf27@%3Cgeneral.incubator.apache.org%3E
>>>
>>> (apologies if these are duplicate links - I re-scanned the thread and
>>> did not spot them)
>>>
>>> Kenn
>>>
>>> On Thu, May 2, 2019 at 8:43 AM Robert Bradshaw <ro...@google.com>
>>> wrote:
>>>
>>>> On Thu, May 2, 2019 at 5:24 PM Michael Luckey <ad...@gmail.com>
>>>> wrote:
>>>> >
>>>> > Thanks Ahmet for calling out to the airflow folks. I believe, I am
>>>> able to follow their argument. So from my point of view I do not have an
>>>> issue with apache policy. But honestly still trying to wrap my head around
>>>> Roberts concern with rebuilding/resigning. Currently, our actual release is
>>>> only a tag on source repo and promoting artefacts. Do not yet understand
>>>> how that needs to change to get PyPi included.
>>>>
>>>> It's not a big change, but let me clarify.
>>>>
>>>> Currently our release preparation goes something like this:
>>>>
>>>> 1) Check out the repo, update the versions to 2.x, build and sign the
>>>> artifacts.
>>>> 2) Announce these artifacts as rcN
>>>> 2a) Push the artifacts to SVN dev/...
>>>> 2b) Push artifacts to the apache maven repository.
>>>> 3) Depending on vote, go back to step (1) or forward to step (4).
>>>> 4) Copy these artifacts as the actual release.
>>>>
>>>> Now if we just try to add (2c) Push these artifacts to Pypi, it will
>>>> be treated (by pypi's tooling, anyone who downloads the tarball, ...)
>>>> as an actual release. You also can't re-push a tarball with the same
>>>> name and different contents (the idea being that named releases should
>>>> never change). So we'd need to change step (1) to update the version
>>>> to 2.x.rcN *and* add a step in (4) to update the version to 2.x (no rc
>>>> suffix), rebuild, resign before publishing.
>>>>
>>>> As mentioned, possibly the rcN suffix could be part of the building
>>>> step for Python.
>>>>
>>>> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <al...@google.com> wrote:
>>>> >>
>>>> >> Michael, Max and other folks who are concerned about the
>>>> compatibility with the apache release policy. Does the information in this
>>>> thread sufficiently address your concerns? Especially the part where, the
>>>> rc artifacts will be protected by a flag (i.e. --pre) from general
>>>> consumption.
>>>> >>
>>>> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw <ro...@google.com>
>>>> wrote:
>>>> >>>
>>>> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay <al...@google.com>
>>>> wrote:
>>>> >>> >
>>>> >>> > This conversation get quite Python centric. Is there a similar
>>>> need for Java?
>>>> >>>
>>>> >>> I think Java is already covered. Go is a different story (but the
>>>> even
>>>> >>> versioning and releasing is being worked out).
>>>> >>>
>>>> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw <
>>>> robertwb@google.com> wrote:
>>>> >>> >>
>>>> >>> >> If we can, by the apache guidelines, post RCs to pypy that is
>>>> >>> >> definitely the way to go. (Note that test.pypi is for developing
>>>> >>> >> against the pypi interface, not for pushing anything real.) The
>>>> caveat
>>>> >>> >> about naming these with rcN in the version number still applies
>>>> >>> >> (that's how pypi guards them against non-explicit installs).
>>>> >>> >
>>>> >>> > Related to the caveat, I believe this can be easily scripted or
>>>> even made part of the travis/wheels pipeline to take the release branch,
>>>> edit the version string in place to add rc, and build the necessary files.
>>>> >>>
>>>> >>> Yes. But the resulting artifacts would have to be rebuilt (and
>>>> >>> re-signed) without the version edit for the actual release. (Well,
>>>> we
>>>> >>> could possibly edit the artifacts rather than rebuild them.) And
>>>> >>> pushing un-edited ones early would be really bad. (It's the classic
>>>> >>> tension of whether a pre-release should be marked internally or
>>>> >>> externally, re-publishing a new set of bits for the actual release
>>>> or
>>>> >>> re-using version numbers for different sets of bits. Pypi does one,
>>>> >>> apache does another...)
>>>> >>>
>>>> >>> >> The advantage is that a user can do "pip install --pre
>>>> apache-beam" to
>>>> >>> >> get the latest rc rather than "pip install
>>>> >>> >>
>>>> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path
>>>> "
>>>> >>> >>
>>>> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada <
>>>> pabloem@google.com> wrote:
>>>> >>> >> >
>>>> >>> >> > Aw that's interesting!
>>>> >>> >> >
>>>> >>> >> > I think, with these considerations, I am only marginally more
>>>> inclined towards publishing to test.pypi. That would make me a +0.9 on
>>>> publishing RCs to the main pip repo then.
>>>> >>> >> >
>>>> >>> >> > Thanks for doing the research Ahmet. :)
>>>> >>> >> > Best
>>>> >>> >> > -P
>>>> >>> >> >
>>>> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <al...@google.com>
>>>> wrote:
>>>> >>> >> >>
>>>> >>> >> >> I asked to Airflow folks about this. See [1] for the full
>>>> response and a link to one of their RC emails. To summarize their position
>>>> (specifically for pypi) is: Unless a user does something explicit (such as
>>>> using a flag, or explicitly requesting an rc release), pip install will not
>>>> serve RC binaries. And that is compatible with RC section of
>>>> http://www.apache.org/legal/release-policy.html#release-types
>>>> >>> >> >>
>>>> >>> >> >> Ahmet
>>>> >>> >> >>
>>>> >>> >> >> [1]
>>>> https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E
>>>> >>> >> >>
>>>> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <al...@google.com>
>>>> wrote:
>>>> >>> >> >>>
>>>> >>> >> >>> The incremental value of publishing python artifacts to a
>>>> separate place but not to actual pypi listing will be low. Users can
>>>> already download RC artifacts, or even pip install from http location
>>>> directly. I think the incremental value will be low, because for a user or
>>>> a downstream library to test with Beam RCs using their usual ways will
>>>> still require them to get other dependencies from the regular pypi listing.
>>>> That would mean they need to change their setup to test with beam rcs,
>>>> which is the same state as today. There will be some incremental value of
>>>> putting them in more obvious places (e.g. pypi test repository). I would
>>>> rather not complicate the release process for doing this.
>>>> >>> >> >>>
>>>> >>> >> >>>
>>>> >>> >> >>>
>>>> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles <
>>>> kenn@apache.org> wrote:
>>>> >>> >> >>>>
>>>> >>> >> >>>> Pip is also able to be pointed at any raw hosted directory
>>>> for the install, right? So we could publish RCs or snapshots somewhere with
>>>> more obvious caveats and not interfere with the pypi list of actual
>>>> releases. Much like the Java snapshots are stored in a separate opt-in
>>>> repository.
>>>> >>> >> >>>>
>>>> >>> >> >>>> Kenn
>>>> >>> >> >>>>
>>>> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels <
>>>> mxm@apache.org> wrote:
>>>> >>> >> >>>>>
>>>> >>> >> >>>>> > wouldn't that be in conflict with Apache release policy
>>>> [1] ?
>>>> >>> >> >>>>> > [1] http://www.apache.org/legal/release-policy.html
>>>> >>> >> >>>>>
>>>> >>> >> >>>>> Indeed, advertising pre-release artifacts is against ASF
>>>> rules. For
>>>> >>> >> >>>>> example, Flink was asked to remove a link to the Maven
>>>> snapshot
>>>> >>> >> >>>>> repository from their download page.
>>>> >>> >> >>>>>
>>>> >>> >> >>>>> However, that does not mean we cannot publish Python
>>>> artifacts. We just
>>>> >>> >> >>>>> have to clearly mark them for developers only and not
>>>> advertise them
>>>> >>> >> >>>>> alongside with the official releases.
>>>> >>> >> >>>>>
>>>> >>> >> >>>>> -Max
>>>> >>> >> >>>>>
>>>> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote:
>>>> >>> >> >>>>> > Don't we push java artifacts to maven repositories as
>>>> part of the RC
>>>> >>> >> >>>>> > process? And completely unvetted snapshots? (Or is this
>>>> OK because
>>>> >>> >> >>>>> > they are special opt-in apache-only ones?)
>>>> >>> >> >>>>> >
>>>> >>> >> >>>>> > I am generally in favor of the idea, but would like to
>>>> avoid increased
>>>> >>> >> >>>>> > toil on the release manager.
>>>> >>> >> >>>>> >
>>>> >>> >> >>>>> > One potential hitch I see is that current release
>>>> process updates the
>>>> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release indicator
>>>> in the version
>>>> >>> >> >>>>> > number) whereas pypi (and other systems) typically
>>>> expect distinct
>>>> >>> >> >>>>> > (recognizable) version numbers for each attempt, and
>>>> only the actual
>>>> >>> >> >>>>> > final result has the actual final release version.
>>>> >>> >> >>>>> >
>>>> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay <
>>>> altay@google.com> wrote:
>>>> >>> >> >>>>> >>
>>>> >>> >> >>>>> >> I do not know the answer.I believe this will be similar
>>>> to sharing the RC artifacts for validation purposes and would not be a
>>>> formal release by itself. But I am not an expert and I hope others will
>>>> share their opinions.
>>>> >>> >> >>>>> >>
>>>> >>> >> >>>>> >> I quickly searched pypi for apache projects and found
>>>> at least airflow [1] and libcloud [2] are publishing rc artifacts to pypi.
>>>> We can reach out to those communities and learn about their processes.
>>>> >>> >> >>>>> >>
>>>> >>> >> >>>>> >> Ahmet
>>>> >>> >> >>>>> >>
>>>> >>> >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history
>>>> >>> >> >>>>> >> [2] https://pypi.org/project/apache-libcloud/#history
>>>> >>> >> >>>>> >>
>>>> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey <
>>>> adude3141@gmail.com> wrote:
>>>> >>> >> >>>>> >>>
>>>> >>> >> >>>>> >>> Hi,
>>>> >>> >> >>>>> >>>
>>>> >>> >> >>>>> >>> wouldn't that be in conflict with Apache release
>>>> policy [1] ?
>>>> >>> >> >>>>> >>>
>>>> >>> >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html
>>>> >>> >> >>>>> >>>
>>>> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold <
>>>> amyrvold@google.com> wrote:
>>>> >>> >> >>>>> >>>>
>>>> >>> >> >>>>> >>>> Great idea. I like the RC candidates to follow as
>>>> much as the release artifact process as possible.
>>>> >>> >> >>>>> >>>>
>>>> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay <
>>>> altay@google.com> wrote:
>>>> >>> >> >>>>> >>>>>
>>>> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing publishing to
>>>> the production pypi repository with an rc tag in the version. And in turn
>>>> allow users to depend on beam's rc version + all the other regular
>>>> dependencies users would have directly from pypi.
>>>> >>> >> >>>>> >>>>>
>>>> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be helpful
>>>> if test pypi repo also mirrors other packages that exist in the production
>>>> pypi repository.
>>>> >>> >> >>>>> >>>>>
>>>> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada <
>>>> pabloem@google.com> wrote:
>>>> >>> >> >>>>> >>>>>>
>>>> >>> >> >>>>> >>>>>> I think this is a great idea. A way of doing it for
>>>> python would be by using the test repository for PyPi[1], and that way we
>>>> would not have to do an official PyPi release, but still would be able to
>>>> install it with pip (by passing an extra flag), and test.
>>>> >>> >> >>>>> >>>>>>
>>>> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts already in
>>>> there[2]. At some point I looked into this, but couldn't figure out who has
>>>> access/the password for it.
>>>> >>> >> >>>>> >>>>>
>>>> >>> >> >>>>> >>>>>
>>>> >>> >> >>>>> >>>>> I also don't know who owns beam package in test pypi
>>>> repo. Does anybody know?
>>>> >>> >> >>>>> >>>>>
>>>> >>> >> >>>>> >>>>>>
>>>> >>> >> >>>>> >>>>>>
>>>> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the test
>>>> PyPi repo to avoid publishing to the main PyPi repo.
>>>> >>> >> >>>>> >>>>>> Best
>>>> >>> >> >>>>> >>>>>> -P.
>>>> >>> >> >>>>> >>>>>>
>>>> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/
>>>> >>> >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/
>>>> >>> >> >>>>> >>>>>>
>>>> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay <
>>>> altay@google.com> wrote:
>>>> >>> >> >>>>> >>>>>>>
>>>> >>> >> >>>>> >>>>>>> Hi all,
>>>> >>> >> >>>>> >>>>>>>
>>>> >>> >> >>>>> >>>>>>> What do you think about the idea of publishing
>>>> pre-release artifacts as part of the RC emails?
>>>> >>> >> >>>>> >>>>>>>
>>>> >>> >> >>>>> >>>>>>> For Python this would translate into publishing
>>>> the same artifacts from RC email with a version like "2.X.0rcY" to pypi. I
>>>> do not know, but I am guessing we can do a similar thing with Maven central
>>>> for Java artifacts as well.
>>>> >>> >> >>>>> >>>>>>>
>>>> >>> >> >>>>> >>>>>>> Advantages would be:
>>>> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their own
>>>> purposes using the same exact process they will normally use.
>>>> >>> >> >>>>> >>>>>>>   - Enable early-adaptors to start using RC
>>>> releases early on in the release cycle if that is what they would like to
>>>> do. This will in turn reduce time pressure on some releases. Especially for
>>>> cases like someone needs a release to be finalized for an upcoming event.
>>>> >>> >> >>>>> >>>>>>>
>>>> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I could
>>>> think of:
>>>> >>> >> >>>>> >>>>>>> - Users could request support for RC artifacts.
>>>> Hopefully in the form of feedback for us to improve the release. But it
>>>> could also be in the form of folks using RC artifacts for production for a
>>>> long time.
>>>> >>> >> >>>>> >>>>>>> - It will add toil to the current release process,
>>>> there will be one more step for each RC. I think for python this will be a
>>>> small step but nevertheless it will be additional work.
>>>> >>> >> >>>>> >>>>>>>
>>>> >>> >> >>>>> >>>>>>> For an example of this, you can take a look at
>>>> tensorflow releases. For 1.13 there were 3 pre-releases [1].
>>>> >>> >> >>>>> >>>>>>>
>>>> >>> >> >>>>> >>>>>>> Ahmet
>>>> >>> >> >>>>> >>>>>>>
>>>> >>> >> >>>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history
>>>>
>>>

Re: [Discuss] Publishing pre-release artifacts to repositories

[Kenneth Knowles <ke...@apache.org>]

Pulling out the relevant pypi bit w.r.t. RCs:


> - Release candidates, nightly or snapshots need to be clearly tagged as
> pre-release on https://pypi.org/project/apache<project>/#history
> - The latest version should not point to an artefact containing unapproved
> code e.g. to a release candidate or snapshot
>

These guidelines were pending assessment by legal & infra. I don't know if
there has been an update. It has been a few months.

Kenn

On Thu, May 2, 2019 at 8:51 AM Kenneth Knowles <ke...@apache.org> wrote:

> Ah, and here's one on general@incubator specifically about RCs:
> https://lists.apache.org/thread.html/c4afcf0807d71f844d912a7e5fe6b481f0779bdcf88ccf9abe50a160@%3Cgeneral.incubator.apache.org%3E
>
> Kenn
>
> On Thu, May 2, 2019 at 8:49 AM Kenneth Knowles <ke...@apache.org> wrote:
>
>> I'd suggest looking for experience beyond Beam and Airflow. I don't see
>> links to some relevant threads.
>>
>> Here's one from legal-discuss@ about binary channels and how they relate
>> to source releases:
>> https://lists.apache.org/thread.html/d578819f1afa6b8fb697ea72083e0fb05e43938a23d6e7bb804069b8@%3Clegal-discuss.apache.org%3E
>>
>> Here's one from the incubator about how RCs and tags relate to ASF
>> release policy:
>> https://lists.apache.org/thread.html/982077ef279e50b260d302d96685e40be6fcabdb0bd43d519621cf27@%3Cgeneral.incubator.apache.org%3E
>>
>> (apologies if these are duplicate links - I re-scanned the thread and did
>> not spot them)
>>
>> Kenn
>>
>> On Thu, May 2, 2019 at 8:43 AM Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> On Thu, May 2, 2019 at 5:24 PM Michael Luckey <ad...@gmail.com>
>>> wrote:
>>> >
>>> > Thanks Ahmet for calling out to the airflow folks. I believe, I am
>>> able to follow their argument. So from my point of view I do not have an
>>> issue with apache policy. But honestly still trying to wrap my head around
>>> Roberts concern with rebuilding/resigning. Currently, our actual release is
>>> only a tag on source repo and promoting artefacts. Do not yet understand
>>> how that needs to change to get PyPi included.
>>>
>>> It's not a big change, but let me clarify.
>>>
>>> Currently our release preparation goes something like this:
>>>
>>> 1) Check out the repo, update the versions to 2.x, build and sign the
>>> artifacts.
>>> 2) Announce these artifacts as rcN
>>> 2a) Push the artifacts to SVN dev/...
>>> 2b) Push artifacts to the apache maven repository.
>>> 3) Depending on vote, go back to step (1) or forward to step (4).
>>> 4) Copy these artifacts as the actual release.
>>>
>>> Now if we just try to add (2c) Push these artifacts to Pypi, it will
>>> be treated (by pypi's tooling, anyone who downloads the tarball, ...)
>>> as an actual release. You also can't re-push a tarball with the same
>>> name and different contents (the idea being that named releases should
>>> never change). So we'd need to change step (1) to update the version
>>> to 2.x.rcN *and* add a step in (4) to update the version to 2.x (no rc
>>> suffix), rebuild, resign before publishing.
>>>
>>> As mentioned, possibly the rcN suffix could be part of the building
>>> step for Python.
>>>
>>> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <al...@google.com> wrote:
>>> >>
>>> >> Michael, Max and other folks who are concerned about the
>>> compatibility with the apache release policy. Does the information in this
>>> thread sufficiently address your concerns? Especially the part where, the
>>> rc artifacts will be protected by a flag (i.e. --pre) from general
>>> consumption.
>>> >>
>>> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw <ro...@google.com>
>>> wrote:
>>> >>>
>>> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay <al...@google.com>
>>> wrote:
>>> >>> >
>>> >>> > This conversation get quite Python centric. Is there a similar
>>> need for Java?
>>> >>>
>>> >>> I think Java is already covered. Go is a different story (but the
>>> even
>>> >>> versioning and releasing is being worked out).
>>> >>>
>>> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw <
>>> robertwb@google.com> wrote:
>>> >>> >>
>>> >>> >> If we can, by the apache guidelines, post RCs to pypy that is
>>> >>> >> definitely the way to go. (Note that test.pypi is for developing
>>> >>> >> against the pypi interface, not for pushing anything real.) The
>>> caveat
>>> >>> >> about naming these with rcN in the version number still applies
>>> >>> >> (that's how pypi guards them against non-explicit installs).
>>> >>> >
>>> >>> > Related to the caveat, I believe this can be easily scripted or
>>> even made part of the travis/wheels pipeline to take the release branch,
>>> edit the version string in place to add rc, and build the necessary files.
>>> >>>
>>> >>> Yes. But the resulting artifacts would have to be rebuilt (and
>>> >>> re-signed) without the version edit for the actual release. (Well, we
>>> >>> could possibly edit the artifacts rather than rebuild them.) And
>>> >>> pushing un-edited ones early would be really bad. (It's the classic
>>> >>> tension of whether a pre-release should be marked internally or
>>> >>> externally, re-publishing a new set of bits for the actual release or
>>> >>> re-using version numbers for different sets of bits. Pypi does one,
>>> >>> apache does another...)
>>> >>>
>>> >>> >> The advantage is that a user can do "pip install --pre
>>> apache-beam" to
>>> >>> >> get the latest rc rather than "pip install
>>> >>> >>
>>> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"
>>> >>> >>
>>> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada <
>>> pabloem@google.com> wrote:
>>> >>> >> >
>>> >>> >> > Aw that's interesting!
>>> >>> >> >
>>> >>> >> > I think, with these considerations, I am only marginally more
>>> inclined towards publishing to test.pypi. That would make me a +0.9 on
>>> publishing RCs to the main pip repo then.
>>> >>> >> >
>>> >>> >> > Thanks for doing the research Ahmet. :)
>>> >>> >> > Best
>>> >>> >> > -P
>>> >>> >> >
>>> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <al...@google.com>
>>> wrote:
>>> >>> >> >>
>>> >>> >> >> I asked to Airflow folks about this. See [1] for the full
>>> response and a link to one of their RC emails. To summarize their position
>>> (specifically for pypi) is: Unless a user does something explicit (such as
>>> using a flag, or explicitly requesting an rc release), pip install will not
>>> serve RC binaries. And that is compatible with RC section of
>>> http://www.apache.org/legal/release-policy.html#release-types
>>> >>> >> >>
>>> >>> >> >> Ahmet
>>> >>> >> >>
>>> >>> >> >> [1]
>>> https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E
>>> >>> >> >>
>>> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <al...@google.com>
>>> wrote:
>>> >>> >> >>>
>>> >>> >> >>> The incremental value of publishing python artifacts to a
>>> separate place but not to actual pypi listing will be low. Users can
>>> already download RC artifacts, or even pip install from http location
>>> directly. I think the incremental value will be low, because for a user or
>>> a downstream library to test with Beam RCs using their usual ways will
>>> still require them to get other dependencies from the regular pypi listing.
>>> That would mean they need to change their setup to test with beam rcs,
>>> which is the same state as today. There will be some incremental value of
>>> putting them in more obvious places (e.g. pypi test repository). I would
>>> rather not complicate the release process for doing this.
>>> >>> >> >>>
>>> >>> >> >>>
>>> >>> >> >>>
>>> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles <
>>> kenn@apache.org> wrote:
>>> >>> >> >>>>
>>> >>> >> >>>> Pip is also able to be pointed at any raw hosted directory
>>> for the install, right? So we could publish RCs or snapshots somewhere with
>>> more obvious caveats and not interfere with the pypi list of actual
>>> releases. Much like the Java snapshots are stored in a separate opt-in
>>> repository.
>>> >>> >> >>>>
>>> >>> >> >>>> Kenn
>>> >>> >> >>>>
>>> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels <
>>> mxm@apache.org> wrote:
>>> >>> >> >>>>>
>>> >>> >> >>>>> > wouldn't that be in conflict with Apache release policy
>>> [1] ?
>>> >>> >> >>>>> > [1] http://www.apache.org/legal/release-policy.html
>>> >>> >> >>>>>
>>> >>> >> >>>>> Indeed, advertising pre-release artifacts is against ASF
>>> rules. For
>>> >>> >> >>>>> example, Flink was asked to remove a link to the Maven
>>> snapshot
>>> >>> >> >>>>> repository from their download page.
>>> >>> >> >>>>>
>>> >>> >> >>>>> However, that does not mean we cannot publish Python
>>> artifacts. We just
>>> >>> >> >>>>> have to clearly mark them for developers only and not
>>> advertise them
>>> >>> >> >>>>> alongside with the official releases.
>>> >>> >> >>>>>
>>> >>> >> >>>>> -Max
>>> >>> >> >>>>>
>>> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote:
>>> >>> >> >>>>> > Don't we push java artifacts to maven repositories as
>>> part of the RC
>>> >>> >> >>>>> > process? And completely unvetted snapshots? (Or is this
>>> OK because
>>> >>> >> >>>>> > they are special opt-in apache-only ones?)
>>> >>> >> >>>>> >
>>> >>> >> >>>>> > I am generally in favor of the idea, but would like to
>>> avoid increased
>>> >>> >> >>>>> > toil on the release manager.
>>> >>> >> >>>>> >
>>> >>> >> >>>>> > One potential hitch I see is that current release process
>>> updates the
>>> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release indicator
>>> in the version
>>> >>> >> >>>>> > number) whereas pypi (and other systems) typically expect
>>> distinct
>>> >>> >> >>>>> > (recognizable) version numbers for each attempt, and only
>>> the actual
>>> >>> >> >>>>> > final result has the actual final release version.
>>> >>> >> >>>>> >
>>> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay <
>>> altay@google.com> wrote:
>>> >>> >> >>>>> >>
>>> >>> >> >>>>> >> I do not know the answer.I believe this will be similar
>>> to sharing the RC artifacts for validation purposes and would not be a
>>> formal release by itself. But I am not an expert and I hope others will
>>> share their opinions.
>>> >>> >> >>>>> >>
>>> >>> >> >>>>> >> I quickly searched pypi for apache projects and found at
>>> least airflow [1] and libcloud [2] are publishing rc artifacts to pypi. We
>>> can reach out to those communities and learn about their processes.
>>> >>> >> >>>>> >>
>>> >>> >> >>>>> >> Ahmet
>>> >>> >> >>>>> >>
>>> >>> >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history
>>> >>> >> >>>>> >> [2] https://pypi.org/project/apache-libcloud/#history
>>> >>> >> >>>>> >>
>>> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey <
>>> adude3141@gmail.com> wrote:
>>> >>> >> >>>>> >>>
>>> >>> >> >>>>> >>> Hi,
>>> >>> >> >>>>> >>>
>>> >>> >> >>>>> >>> wouldn't that be in conflict with Apache release policy
>>> [1] ?
>>> >>> >> >>>>> >>>
>>> >>> >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html
>>> >>> >> >>>>> >>>
>>> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold <
>>> amyrvold@google.com> wrote:
>>> >>> >> >>>>> >>>>
>>> >>> >> >>>>> >>>> Great idea. I like the RC candidates to follow as much
>>> as the release artifact process as possible.
>>> >>> >> >>>>> >>>>
>>> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay <
>>> altay@google.com> wrote:
>>> >>> >> >>>>> >>>>>
>>> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing publishing to
>>> the production pypi repository with an rc tag in the version. And in turn
>>> allow users to depend on beam's rc version + all the other regular
>>> dependencies users would have directly from pypi.
>>> >>> >> >>>>> >>>>>
>>> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be helpful if
>>> test pypi repo also mirrors other packages that exist in the production
>>> pypi repository.
>>> >>> >> >>>>> >>>>>
>>> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada <
>>> pabloem@google.com> wrote:
>>> >>> >> >>>>> >>>>>>
>>> >>> >> >>>>> >>>>>> I think this is a great idea. A way of doing it for
>>> python would be by using the test repository for PyPi[1], and that way we
>>> would not have to do an official PyPi release, but still would be able to
>>> install it with pip (by passing an extra flag), and test.
>>> >>> >> >>>>> >>>>>>
>>> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts already in
>>> there[2]. At some point I looked into this, but couldn't figure out who has
>>> access/the password for it.
>>> >>> >> >>>>> >>>>>
>>> >>> >> >>>>> >>>>>
>>> >>> >> >>>>> >>>>> I also don't know who owns beam package in test pypi
>>> repo. Does anybody know?
>>> >>> >> >>>>> >>>>>
>>> >>> >> >>>>> >>>>>>
>>> >>> >> >>>>> >>>>>>
>>> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the test
>>> PyPi repo to avoid publishing to the main PyPi repo.
>>> >>> >> >>>>> >>>>>> Best
>>> >>> >> >>>>> >>>>>> -P.
>>> >>> >> >>>>> >>>>>>
>>> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/
>>> >>> >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/
>>> >>> >> >>>>> >>>>>>
>>> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay <
>>> altay@google.com> wrote:
>>> >>> >> >>>>> >>>>>>>
>>> >>> >> >>>>> >>>>>>> Hi all,
>>> >>> >> >>>>> >>>>>>>
>>> >>> >> >>>>> >>>>>>> What do you think about the idea of publishing
>>> pre-release artifacts as part of the RC emails?
>>> >>> >> >>>>> >>>>>>>
>>> >>> >> >>>>> >>>>>>> For Python this would translate into publishing the
>>> same artifacts from RC email with a version like "2.X.0rcY" to pypi. I do
>>> not know, but I am guessing we can do a similar thing with Maven central
>>> for Java artifacts as well.
>>> >>> >> >>>>> >>>>>>>
>>> >>> >> >>>>> >>>>>>> Advantages would be:
>>> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their own
>>> purposes using the same exact process they will normally use.
>>> >>> >> >>>>> >>>>>>>   - Enable early-adaptors to start using RC
>>> releases early on in the release cycle if that is what they would like to
>>> do. This will in turn reduce time pressure on some releases. Especially for
>>> cases like someone needs a release to be finalized for an upcoming event.
>>> >>> >> >>>>> >>>>>>>
>>> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I could
>>> think of:
>>> >>> >> >>>>> >>>>>>> - Users could request support for RC artifacts.
>>> Hopefully in the form of feedback for us to improve the release. But it
>>> could also be in the form of folks using RC artifacts for production for a
>>> long time.
>>> >>> >> >>>>> >>>>>>> - It will add toil to the current release process,
>>> there will be one more step for each RC. I think for python this will be a
>>> small step but nevertheless it will be additional work.
>>> >>> >> >>>>> >>>>>>>
>>> >>> >> >>>>> >>>>>>> For an example of this, you can take a look at
>>> tensorflow releases. For 1.13 there were 3 pre-releases [1].
>>> >>> >> >>>>> >>>>>>>
>>> >>> >> >>>>> >>>>>>> Ahmet
>>> >>> >> >>>>> >>>>>>>
>>> >>> >> >>>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history
>>>
>>

Re: [Discuss] Publishing pre-release artifacts to repositories

[Kenneth Knowles <ke...@apache.org>]

Ah, and here's one on general@incubator specifically about RCs:
https://lists.apache.org/thread.html/c4afcf0807d71f844d912a7e5fe6b481f0779bdcf88ccf9abe50a160@%3Cgeneral.incubator.apache.org%3E

Kenn

On Thu, May 2, 2019 at 8:49 AM Kenneth Knowles <ke...@apache.org> wrote:

> I'd suggest looking for experience beyond Beam and Airflow. I don't see
> links to some relevant threads.
>
> Here's one from legal-discuss@ about binary channels and how they relate
> to source releases:
> https://lists.apache.org/thread.html/d578819f1afa6b8fb697ea72083e0fb05e43938a23d6e7bb804069b8@%3Clegal-discuss.apache.org%3E
>
> Here's one from the incubator about how RCs and tags relate to ASF release
> policy:
> https://lists.apache.org/thread.html/982077ef279e50b260d302d96685e40be6fcabdb0bd43d519621cf27@%3Cgeneral.incubator.apache.org%3E
>
> (apologies if these are duplicate links - I re-scanned the thread and did
> not spot them)
>
> Kenn
>
> On Thu, May 2, 2019 at 8:43 AM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> On Thu, May 2, 2019 at 5:24 PM Michael Luckey <ad...@gmail.com>
>> wrote:
>> >
>> > Thanks Ahmet for calling out to the airflow folks. I believe, I am able
>> to follow their argument. So from my point of view I do not have an issue
>> with apache policy. But honestly still trying to wrap my head around
>> Roberts concern with rebuilding/resigning. Currently, our actual release is
>> only a tag on source repo and promoting artefacts. Do not yet understand
>> how that needs to change to get PyPi included.
>>
>> It's not a big change, but let me clarify.
>>
>> Currently our release preparation goes something like this:
>>
>> 1) Check out the repo, update the versions to 2.x, build and sign the
>> artifacts.
>> 2) Announce these artifacts as rcN
>> 2a) Push the artifacts to SVN dev/...
>> 2b) Push artifacts to the apache maven repository.
>> 3) Depending on vote, go back to step (1) or forward to step (4).
>> 4) Copy these artifacts as the actual release.
>>
>> Now if we just try to add (2c) Push these artifacts to Pypi, it will
>> be treated (by pypi's tooling, anyone who downloads the tarball, ...)
>> as an actual release. You also can't re-push a tarball with the same
>> name and different contents (the idea being that named releases should
>> never change). So we'd need to change step (1) to update the version
>> to 2.x.rcN *and* add a step in (4) to update the version to 2.x (no rc
>> suffix), rebuild, resign before publishing.
>>
>> As mentioned, possibly the rcN suffix could be part of the building
>> step for Python.
>>
>> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <al...@google.com> wrote:
>> >>
>> >> Michael, Max and other folks who are concerned about the compatibility
>> with the apache release policy. Does the information in this thread
>> sufficiently address your concerns? Especially the part where, the rc
>> artifacts will be protected by a flag (i.e. --pre) from general consumption.
>> >>
>> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>> >>>
>> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay <al...@google.com> wrote:
>> >>> >
>> >>> > This conversation get quite Python centric. Is there a similar need
>> for Java?
>> >>>
>> >>> I think Java is already covered. Go is a different story (but the even
>> >>> versioning and releasing is being worked out).
>> >>>
>> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw <
>> robertwb@google.com> wrote:
>> >>> >>
>> >>> >> If we can, by the apache guidelines, post RCs to pypy that is
>> >>> >> definitely the way to go. (Note that test.pypi is for developing
>> >>> >> against the pypi interface, not for pushing anything real.) The
>> caveat
>> >>> >> about naming these with rcN in the version number still applies
>> >>> >> (that's how pypi guards them against non-explicit installs).
>> >>> >
>> >>> > Related to the caveat, I believe this can be easily scripted or
>> even made part of the travis/wheels pipeline to take the release branch,
>> edit the version string in place to add rc, and build the necessary files.
>> >>>
>> >>> Yes. But the resulting artifacts would have to be rebuilt (and
>> >>> re-signed) without the version edit for the actual release. (Well, we
>> >>> could possibly edit the artifacts rather than rebuild them.) And
>> >>> pushing un-edited ones early would be really bad. (It's the classic
>> >>> tension of whether a pre-release should be marked internally or
>> >>> externally, re-publishing a new set of bits for the actual release or
>> >>> re-using version numbers for different sets of bits. Pypi does one,
>> >>> apache does another...)
>> >>>
>> >>> >> The advantage is that a user can do "pip install --pre
>> apache-beam" to
>> >>> >> get the latest rc rather than "pip install
>> >>> >>
>> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"
>> >>> >>
>> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada <pa...@google.com>
>> wrote:
>> >>> >> >
>> >>> >> > Aw that's interesting!
>> >>> >> >
>> >>> >> > I think, with these considerations, I am only marginally more
>> inclined towards publishing to test.pypi. That would make me a +0.9 on
>> publishing RCs to the main pip repo then.
>> >>> >> >
>> >>> >> > Thanks for doing the research Ahmet. :)
>> >>> >> > Best
>> >>> >> > -P
>> >>> >> >
>> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <al...@google.com>
>> wrote:
>> >>> >> >>
>> >>> >> >> I asked to Airflow folks about this. See [1] for the full
>> response and a link to one of their RC emails. To summarize their position
>> (specifically for pypi) is: Unless a user does something explicit (such as
>> using a flag, or explicitly requesting an rc release), pip install will not
>> serve RC binaries. And that is compatible with RC section of
>> http://www.apache.org/legal/release-policy.html#release-types
>> >>> >> >>
>> >>> >> >> Ahmet
>> >>> >> >>
>> >>> >> >> [1]
>> https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E
>> >>> >> >>
>> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <al...@google.com>
>> wrote:
>> >>> >> >>>
>> >>> >> >>> The incremental value of publishing python artifacts to a
>> separate place but not to actual pypi listing will be low. Users can
>> already download RC artifacts, or even pip install from http location
>> directly. I think the incremental value will be low, because for a user or
>> a downstream library to test with Beam RCs using their usual ways will
>> still require them to get other dependencies from the regular pypi listing.
>> That would mean they need to change their setup to test with beam rcs,
>> which is the same state as today. There will be some incremental value of
>> putting them in more obvious places (e.g. pypi test repository). I would
>> rather not complicate the release process for doing this.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles <
>> kenn@apache.org> wrote:
>> >>> >> >>>>
>> >>> >> >>>> Pip is also able to be pointed at any raw hosted directory
>> for the install, right? So we could publish RCs or snapshots somewhere with
>> more obvious caveats and not interfere with the pypi list of actual
>> releases. Much like the Java snapshots are stored in a separate opt-in
>> repository.
>> >>> >> >>>>
>> >>> >> >>>> Kenn
>> >>> >> >>>>
>> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels <
>> mxm@apache.org> wrote:
>> >>> >> >>>>>
>> >>> >> >>>>> > wouldn't that be in conflict with Apache release policy
>> [1] ?
>> >>> >> >>>>> > [1] http://www.apache.org/legal/release-policy.html
>> >>> >> >>>>>
>> >>> >> >>>>> Indeed, advertising pre-release artifacts is against ASF
>> rules. For
>> >>> >> >>>>> example, Flink was asked to remove a link to the Maven
>> snapshot
>> >>> >> >>>>> repository from their download page.
>> >>> >> >>>>>
>> >>> >> >>>>> However, that does not mean we cannot publish Python
>> artifacts. We just
>> >>> >> >>>>> have to clearly mark them for developers only and not
>> advertise them
>> >>> >> >>>>> alongside with the official releases.
>> >>> >> >>>>>
>> >>> >> >>>>> -Max
>> >>> >> >>>>>
>> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote:
>> >>> >> >>>>> > Don't we push java artifacts to maven repositories as part
>> of the RC
>> >>> >> >>>>> > process? And completely unvetted snapshots? (Or is this OK
>> because
>> >>> >> >>>>> > they are special opt-in apache-only ones?)
>> >>> >> >>>>> >
>> >>> >> >>>>> > I am generally in favor of the idea, but would like to
>> avoid increased
>> >>> >> >>>>> > toil on the release manager.
>> >>> >> >>>>> >
>> >>> >> >>>>> > One potential hitch I see is that current release process
>> updates the
>> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release indicator in
>> the version
>> >>> >> >>>>> > number) whereas pypi (and other systems) typically expect
>> distinct
>> >>> >> >>>>> > (recognizable) version numbers for each attempt, and only
>> the actual
>> >>> >> >>>>> > final result has the actual final release version.
>> >>> >> >>>>> >
>> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay <
>> altay@google.com> wrote:
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> I do not know the answer.I believe this will be similar
>> to sharing the RC artifacts for validation purposes and would not be a
>> formal release by itself. But I am not an expert and I hope others will
>> share their opinions.
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> I quickly searched pypi for apache projects and found at
>> least airflow [1] and libcloud [2] are publishing rc artifacts to pypi. We
>> can reach out to those communities and learn about their processes.
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> Ahmet
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history
>> >>> >> >>>>> >> [2] https://pypi.org/project/apache-libcloud/#history
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey <
>> adude3141@gmail.com> wrote:
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> Hi,
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> wouldn't that be in conflict with Apache release policy
>> [1] ?
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold <
>> amyrvold@google.com> wrote:
>> >>> >> >>>>> >>>>
>> >>> >> >>>>> >>>> Great idea. I like the RC candidates to follow as much
>> as the release artifact process as possible.
>> >>> >> >>>>> >>>>
>> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay <
>> altay@google.com> wrote:
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing publishing to
>> the production pypi repository with an rc tag in the version. And in turn
>> allow users to depend on beam's rc version + all the other regular
>> dependencies users would have directly from pypi.
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be helpful if
>> test pypi repo also mirrors other packages that exist in the production
>> pypi repository.
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada <
>> pabloem@google.com> wrote:
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> I think this is a great idea. A way of doing it for
>> python would be by using the test repository for PyPi[1], and that way we
>> would not have to do an official PyPi release, but still would be able to
>> install it with pip (by passing an extra flag), and test.
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts already in
>> there[2]. At some point I looked into this, but couldn't figure out who has
>> access/the password for it.
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>> I also don't know who owns beam package in test pypi
>> repo. Does anybody know?
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the test PyPi
>> repo to avoid publishing to the main PyPi repo.
>> >>> >> >>>>> >>>>>> Best
>> >>> >> >>>>> >>>>>> -P.
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/
>> >>> >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay <
>> altay@google.com> wrote:
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> Hi all,
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> What do you think about the idea of publishing
>> pre-release artifacts as part of the RC emails?
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> For Python this would translate into publishing the
>> same artifacts from RC email with a version like "2.X.0rcY" to pypi. I do
>> not know, but I am guessing we can do a similar thing with Maven central
>> for Java artifacts as well.
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> Advantages would be:
>> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their own
>> purposes using the same exact process they will normally use.
>> >>> >> >>>>> >>>>>>>   - Enable early-adaptors to start using RC releases
>> early on in the release cycle if that is what they would like to do. This
>> will in turn reduce time pressure on some releases. Especially for cases
>> like someone needs a release to be finalized for an upcoming event.
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I could think
>> of:
>> >>> >> >>>>> >>>>>>> - Users could request support for RC artifacts.
>> Hopefully in the form of feedback for us to improve the release. But it
>> could also be in the form of folks using RC artifacts for production for a
>> long time.
>> >>> >> >>>>> >>>>>>> - It will add toil to the current release process,
>> there will be one more step for each RC. I think for python this will be a
>> small step but nevertheless it will be additional work.
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> For an example of this, you can take a look at
>> tensorflow releases. For 1.13 there were 3 pre-releases [1].
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> Ahmet
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history
>>
>

Re: [Discuss] Publishing pre-release artifacts to repositories

[Kenneth Knowles <ke...@apache.org>]

I'd suggest looking for experience beyond Beam and Airflow. I don't see
links to some relevant threads.

Here's one from legal-discuss@ about binary channels and how they relate to
source releases:
https://lists.apache.org/thread.html/d578819f1afa6b8fb697ea72083e0fb05e43938a23d6e7bb804069b8@%3Clegal-discuss.apache.org%3E

Here's one from the incubator about how RCs and tags relate to ASF release
policy:
https://lists.apache.org/thread.html/982077ef279e50b260d302d96685e40be6fcabdb0bd43d519621cf27@%3Cgeneral.incubator.apache.org%3E

(apologies if these are duplicate links - I re-scanned the thread and did
not spot them)

Kenn

On Thu, May 2, 2019 at 8:43 AM Robert Bradshaw <ro...@google.com> wrote:

> On Thu, May 2, 2019 at 5:24 PM Michael Luckey <ad...@gmail.com> wrote:
> >
> > Thanks Ahmet for calling out to the airflow folks. I believe, I am able
> to follow their argument. So from my point of view I do not have an issue
> with apache policy. But honestly still trying to wrap my head around
> Roberts concern with rebuilding/resigning. Currently, our actual release is
> only a tag on source repo and promoting artefacts. Do not yet understand
> how that needs to change to get PyPi included.
>
> It's not a big change, but let me clarify.
>
> Currently our release preparation goes something like this:
>
> 1) Check out the repo, update the versions to 2.x, build and sign the
> artifacts.
> 2) Announce these artifacts as rcN
> 2a) Push the artifacts to SVN dev/...
> 2b) Push artifacts to the apache maven repository.
> 3) Depending on vote, go back to step (1) or forward to step (4).
> 4) Copy these artifacts as the actual release.
>
> Now if we just try to add (2c) Push these artifacts to Pypi, it will
> be treated (by pypi's tooling, anyone who downloads the tarball, ...)
> as an actual release. You also can't re-push a tarball with the same
> name and different contents (the idea being that named releases should
> never change). So we'd need to change step (1) to update the version
> to 2.x.rcN *and* add a step in (4) to update the version to 2.x (no rc
> suffix), rebuild, resign before publishing.
>
> As mentioned, possibly the rcN suffix could be part of the building
> step for Python.
>
> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <al...@google.com> wrote:
> >>
> >> Michael, Max and other folks who are concerned about the compatibility
> with the apache release policy. Does the information in this thread
> sufficiently address your concerns? Especially the part where, the rc
> artifacts will be protected by a flag (i.e. --pre) from general consumption.
> >>
> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>>
> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay <al...@google.com> wrote:
> >>> >
> >>> > This conversation get quite Python centric. Is there a similar need
> for Java?
> >>>
> >>> I think Java is already covered. Go is a different story (but the even
> >>> versioning and releasing is being worked out).
> >>>
> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw <ro...@google.com>
> wrote:
> >>> >>
> >>> >> If we can, by the apache guidelines, post RCs to pypy that is
> >>> >> definitely the way to go. (Note that test.pypi is for developing
> >>> >> against the pypi interface, not for pushing anything real.) The
> caveat
> >>> >> about naming these with rcN in the version number still applies
> >>> >> (that's how pypi guards them against non-explicit installs).
> >>> >
> >>> > Related to the caveat, I believe this can be easily scripted or even
> made part of the travis/wheels pipeline to take the release branch, edit
> the version string in place to add rc, and build the necessary files.
> >>>
> >>> Yes. But the resulting artifacts would have to be rebuilt (and
> >>> re-signed) without the version edit for the actual release. (Well, we
> >>> could possibly edit the artifacts rather than rebuild them.) And
> >>> pushing un-edited ones early would be really bad. (It's the classic
> >>> tension of whether a pre-release should be marked internally or
> >>> externally, re-publishing a new set of bits for the actual release or
> >>> re-using version numbers for different sets of bits. Pypi does one,
> >>> apache does another...)
> >>>
> >>> >> The advantage is that a user can do "pip install --pre apache-beam"
> to
> >>> >> get the latest rc rather than "pip install
> >>> >>
> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"
> >>> >>
> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada <pa...@google.com>
> wrote:
> >>> >> >
> >>> >> > Aw that's interesting!
> >>> >> >
> >>> >> > I think, with these considerations, I am only marginally more
> inclined towards publishing to test.pypi. That would make me a +0.9 on
> publishing RCs to the main pip repo then.
> >>> >> >
> >>> >> > Thanks for doing the research Ahmet. :)
> >>> >> > Best
> >>> >> > -P
> >>> >> >
> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <al...@google.com>
> wrote:
> >>> >> >>
> >>> >> >> I asked to Airflow folks about this. See [1] for the full
> response and a link to one of their RC emails. To summarize their position
> (specifically for pypi) is: Unless a user does something explicit (such as
> using a flag, or explicitly requesting an rc release), pip install will not
> serve RC binaries. And that is compatible with RC section of
> http://www.apache.org/legal/release-policy.html#release-types
> >>> >> >>
> >>> >> >> Ahmet
> >>> >> >>
> >>> >> >> [1]
> https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E
> >>> >> >>
> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <al...@google.com>
> wrote:
> >>> >> >>>
> >>> >> >>> The incremental value of publishing python artifacts to a
> separate place but not to actual pypi listing will be low. Users can
> already download RC artifacts, or even pip install from http location
> directly. I think the incremental value will be low, because for a user or
> a downstream library to test with Beam RCs using their usual ways will
> still require them to get other dependencies from the regular pypi listing.
> That would mean they need to change their setup to test with beam rcs,
> which is the same state as today. There will be some incremental value of
> putting them in more obvious places (e.g. pypi test repository). I would
> rather not complicate the release process for doing this.
> >>> >> >>>
> >>> >> >>>
> >>> >> >>>
> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles <
> kenn@apache.org> wrote:
> >>> >> >>>>
> >>> >> >>>> Pip is also able to be pointed at any raw hosted directory for
> the install, right? So we could publish RCs or snapshots somewhere with
> more obvious caveats and not interfere with the pypi list of actual
> releases. Much like the Java snapshots are stored in a separate opt-in
> repository.
> >>> >> >>>>
> >>> >> >>>> Kenn
> >>> >> >>>>
> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels <
> mxm@apache.org> wrote:
> >>> >> >>>>>
> >>> >> >>>>> > wouldn't that be in conflict with Apache release policy [1]
> ?
> >>> >> >>>>> > [1] http://www.apache.org/legal/release-policy.html
> >>> >> >>>>>
> >>> >> >>>>> Indeed, advertising pre-release artifacts is against ASF
> rules. For
> >>> >> >>>>> example, Flink was asked to remove a link to the Maven
> snapshot
> >>> >> >>>>> repository from their download page.
> >>> >> >>>>>
> >>> >> >>>>> However, that does not mean we cannot publish Python
> artifacts. We just
> >>> >> >>>>> have to clearly mark them for developers only and not
> advertise them
> >>> >> >>>>> alongside with the official releases.
> >>> >> >>>>>
> >>> >> >>>>> -Max
> >>> >> >>>>>
> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote:
> >>> >> >>>>> > Don't we push java artifacts to maven repositories as part
> of the RC
> >>> >> >>>>> > process? And completely unvetted snapshots? (Or is this OK
> because
> >>> >> >>>>> > they are special opt-in apache-only ones?)
> >>> >> >>>>> >
> >>> >> >>>>> > I am generally in favor of the idea, but would like to
> avoid increased
> >>> >> >>>>> > toil on the release manager.
> >>> >> >>>>> >
> >>> >> >>>>> > One potential hitch I see is that current release process
> updates the
> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release indicator in
> the version
> >>> >> >>>>> > number) whereas pypi (and other systems) typically expect
> distinct
> >>> >> >>>>> > (recognizable) version numbers for each attempt, and only
> the actual
> >>> >> >>>>> > final result has the actual final release version.
> >>> >> >>>>> >
> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay <
> altay@google.com> wrote:
> >>> >> >>>>> >>
> >>> >> >>>>> >> I do not know the answer.I believe this will be similar to
> sharing the RC artifacts for validation purposes and would not be a formal
> release by itself. But I am not an expert and I hope others will share
> their opinions.
> >>> >> >>>>> >>
> >>> >> >>>>> >> I quickly searched pypi for apache projects and found at
> least airflow [1] and libcloud [2] are publishing rc artifacts to pypi. We
> can reach out to those communities and learn about their processes.
> >>> >> >>>>> >>
> >>> >> >>>>> >> Ahmet
> >>> >> >>>>> >>
> >>> >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history
> >>> >> >>>>> >> [2] https://pypi.org/project/apache-libcloud/#history
> >>> >> >>>>> >>
> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey <
> adude3141@gmail.com> wrote:
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> Hi,
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> wouldn't that be in conflict with Apache release policy
> [1] ?
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html
> >>> >> >>>>> >>>
> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold <
> amyrvold@google.com> wrote:
> >>> >> >>>>> >>>>
> >>> >> >>>>> >>>> Great idea. I like the RC candidates to follow as much
> as the release artifact process as possible.
> >>> >> >>>>> >>>>
> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay <
> altay@google.com> wrote:
> >>> >> >>>>> >>>>>
> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing publishing to
> the production pypi repository with an rc tag in the version. And in turn
> allow users to depend on beam's rc version + all the other regular
> dependencies users would have directly from pypi.
> >>> >> >>>>> >>>>>
> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be helpful if
> test pypi repo also mirrors other packages that exist in the production
> pypi repository.
> >>> >> >>>>> >>>>>
> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada <
> pabloem@google.com> wrote:
> >>> >> >>>>> >>>>>>
> >>> >> >>>>> >>>>>> I think this is a great idea. A way of doing it for
> python would be by using the test repository for PyPi[1], and that way we
> would not have to do an official PyPi release, but still would be able to
> install it with pip (by passing an extra flag), and test.
> >>> >> >>>>> >>>>>>
> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts already in
> there[2]. At some point I looked into this, but couldn't figure out who has
> access/the password for it.
> >>> >> >>>>> >>>>>
> >>> >> >>>>> >>>>>
> >>> >> >>>>> >>>>> I also don't know who owns beam package in test pypi
> repo. Does anybody know?
> >>> >> >>>>> >>>>>
> >>> >> >>>>> >>>>>>
> >>> >> >>>>> >>>>>>
> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the test PyPi
> repo to avoid publishing to the main PyPi repo.
> >>> >> >>>>> >>>>>> Best
> >>> >> >>>>> >>>>>> -P.
> >>> >> >>>>> >>>>>>
> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/
> >>> >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/
> >>> >> >>>>> >>>>>>
> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay <
> altay@google.com> wrote:
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> Hi all,
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> What do you think about the idea of publishing
> pre-release artifacts as part of the RC emails?
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> For Python this would translate into publishing the
> same artifacts from RC email with a version like "2.X.0rcY" to pypi. I do
> not know, but I am guessing we can do a similar thing with Maven central
> for Java artifacts as well.
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> Advantages would be:
> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their own
> purposes using the same exact process they will normally use.
> >>> >> >>>>> >>>>>>>   - Enable early-adaptors to start using RC releases
> early on in the release cycle if that is what they would like to do. This
> will in turn reduce time pressure on some releases. Especially for cases
> like someone needs a release to be finalized for an upcoming event.
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I could think
> of:
> >>> >> >>>>> >>>>>>> - Users could request support for RC artifacts.
> Hopefully in the form of feedback for us to improve the release. But it
> could also be in the form of folks using RC artifacts for production for a
> long time.
> >>> >> >>>>> >>>>>>> - It will add toil to the current release process,
> there will be one more step for each RC. I think for python this will be a
> small step but nevertheless it will be additional work.
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> For an example of this, you can take a look at
> tensorflow releases. For 1.13 there were 3 pre-releases [1].
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> Ahmet
> >>> >> >>>>> >>>>>>>
> >>> >> >>>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history
>

Re: [Discuss] Publishing pre-release artifacts to repositories

[Robert Bradshaw <ro...@google.com>]

On Thu, May 2, 2019 at 5:24 PM Michael Luckey <ad...@gmail.com> wrote:
>
> Thanks Ahmet for calling out to the airflow folks. I believe, I am able to follow their argument. So from my point of view I do not have an issue with apache policy. But honestly still trying to wrap my head around Roberts concern with rebuilding/resigning. Currently, our actual release is only a tag on source repo and promoting artefacts. Do not yet understand how that needs to change to get PyPi included.

It's not a big change, but let me clarify.

Currently our release preparation goes something like this:

1) Check out the repo, update the versions to 2.x, build and sign the artifacts.
2) Announce these artifacts as rcN
2a) Push the artifacts to SVN dev/...
2b) Push artifacts to the apache maven repository.
3) Depending on vote, go back to step (1) or forward to step (4).
4) Copy these artifacts as the actual release.

Now if we just try to add (2c) Push these artifacts to Pypi, it will
be treated (by pypi's tooling, anyone who downloads the tarball, ...)
as an actual release. You also can't re-push a tarball with the same
name and different contents (the idea being that named releases should
never change). So we'd need to change step (1) to update the version
to 2.x.rcN *and* add a step in (4) to update the version to 2.x (no rc
suffix), rebuild, resign before publishing.

As mentioned, possibly the rcN suffix could be part of the building
step for Python.

> On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <al...@google.com> wrote:
>>
>> Michael, Max and other folks who are concerned about the compatibility with the apache release policy. Does the information in this thread sufficiently address your concerns? Especially the part where, the rc artifacts will be protected by a flag (i.e. --pre) from general consumption.
>>
>> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw <ro...@google.com> wrote:
>>>
>>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay <al...@google.com> wrote:
>>> >
>>> > This conversation get quite Python centric. Is there a similar need for Java?
>>>
>>> I think Java is already covered. Go is a different story (but the even
>>> versioning and releasing is being worked out).
>>>
>>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw <ro...@google.com> wrote:
>>> >>
>>> >> If we can, by the apache guidelines, post RCs to pypy that is
>>> >> definitely the way to go. (Note that test.pypi is for developing
>>> >> against the pypi interface, not for pushing anything real.) The caveat
>>> >> about naming these with rcN in the version number still applies
>>> >> (that's how pypi guards them against non-explicit installs).
>>> >
>>> > Related to the caveat, I believe this can be easily scripted or even made part of the travis/wheels pipeline to take the release branch, edit the version string in place to add rc, and build the necessary files.
>>>
>>> Yes. But the resulting artifacts would have to be rebuilt (and
>>> re-signed) without the version edit for the actual release. (Well, we
>>> could possibly edit the artifacts rather than rebuild them.) And
>>> pushing un-edited ones early would be really bad. (It's the classic
>>> tension of whether a pre-release should be marked internally or
>>> externally, re-publishing a new set of bits for the actual release or
>>> re-using version numbers for different sets of bits. Pypi does one,
>>> apache does another...)
>>>
>>> >> The advantage is that a user can do "pip install --pre apache-beam" to
>>> >> get the latest rc rather than "pip install
>>> >> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"
>>> >>
>>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada <pa...@google.com> wrote:
>>> >> >
>>> >> > Aw that's interesting!
>>> >> >
>>> >> > I think, with these considerations, I am only marginally more inclined towards publishing to test.pypi. That would make me a +0.9 on publishing RCs to the main pip repo then.
>>> >> >
>>> >> > Thanks for doing the research Ahmet. :)
>>> >> > Best
>>> >> > -P
>>> >> >
>>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <al...@google.com> wrote:
>>> >> >>
>>> >> >> I asked to Airflow folks about this. See [1] for the full response and a link to one of their RC emails. To summarize their position (specifically for pypi) is: Unless a user does something explicit (such as using a flag, or explicitly requesting an rc release), pip install will not serve RC binaries. And that is compatible with RC section of http://www.apache.org/legal/release-policy.html#release-types
>>> >> >>
>>> >> >> Ahmet
>>> >> >>
>>> >> >> [1] https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E
>>> >> >>
>>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <al...@google.com> wrote:
>>> >> >>>
>>> >> >>> The incremental value of publishing python artifacts to a separate place but not to actual pypi listing will be low. Users can already download RC artifacts, or even pip install from http location directly. I think the incremental value will be low, because for a user or a downstream library to test with Beam RCs using their usual ways will still require them to get other dependencies from the regular pypi listing. That would mean they need to change their setup to test with beam rcs, which is the same state as today. There will be some incremental value of putting them in more obvious places (e.g. pypi test repository). I would rather not complicate the release process for doing this.
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles <ke...@apache.org> wrote:
>>> >> >>>>
>>> >> >>>> Pip is also able to be pointed at any raw hosted directory for the install, right? So we could publish RCs or snapshots somewhere with more obvious caveats and not interfere with the pypi list of actual releases. Much like the Java snapshots are stored in a separate opt-in repository.
>>> >> >>>>
>>> >> >>>> Kenn
>>> >> >>>>
>>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels <mx...@apache.org> wrote:
>>> >> >>>>>
>>> >> >>>>> > wouldn't that be in conflict with Apache release policy [1] ?
>>> >> >>>>> > [1] http://www.apache.org/legal/release-policy.html
>>> >> >>>>>
>>> >> >>>>> Indeed, advertising pre-release artifacts is against ASF rules. For
>>> >> >>>>> example, Flink was asked to remove a link to the Maven snapshot
>>> >> >>>>> repository from their download page.
>>> >> >>>>>
>>> >> >>>>> However, that does not mean we cannot publish Python artifacts. We just
>>> >> >>>>> have to clearly mark them for developers only and not advertise them
>>> >> >>>>> alongside with the official releases.
>>> >> >>>>>
>>> >> >>>>> -Max
>>> >> >>>>>
>>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote:
>>> >> >>>>> > Don't we push java artifacts to maven repositories as part of the RC
>>> >> >>>>> > process? And completely unvetted snapshots? (Or is this OK because
>>> >> >>>>> > they are special opt-in apache-only ones?)
>>> >> >>>>> >
>>> >> >>>>> > I am generally in favor of the idea, but would like to avoid increased
>>> >> >>>>> > toil on the release manager.
>>> >> >>>>> >
>>> >> >>>>> > One potential hitch I see is that current release process updates the
>>> >> >>>>> > versions to x.y.z (no RC or other pre-release indicator in the version
>>> >> >>>>> > number) whereas pypi (and other systems) typically expect distinct
>>> >> >>>>> > (recognizable) version numbers for each attempt, and only the actual
>>> >> >>>>> > final result has the actual final release version.
>>> >> >>>>> >
>>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay <al...@google.com> wrote:
>>> >> >>>>> >>
>>> >> >>>>> >> I do not know the answer.I believe this will be similar to sharing the RC artifacts for validation purposes and would not be a formal release by itself. But I am not an expert and I hope others will share their opinions.
>>> >> >>>>> >>
>>> >> >>>>> >> I quickly searched pypi for apache projects and found at least airflow [1] and libcloud [2] are publishing rc artifacts to pypi. We can reach out to those communities and learn about their processes.
>>> >> >>>>> >>
>>> >> >>>>> >> Ahmet
>>> >> >>>>> >>
>>> >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history
>>> >> >>>>> >> [2] https://pypi.org/project/apache-libcloud/#history
>>> >> >>>>> >>
>>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey <ad...@gmail.com> wrote:
>>> >> >>>>> >>>
>>> >> >>>>> >>> Hi,
>>> >> >>>>> >>>
>>> >> >>>>> >>> wouldn't that be in conflict with Apache release policy [1] ?
>>> >> >>>>> >>>
>>> >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html
>>> >> >>>>> >>>
>>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold <am...@google.com> wrote:
>>> >> >>>>> >>>>
>>> >> >>>>> >>>> Great idea. I like the RC candidates to follow as much as the release artifact process as possible.
>>> >> >>>>> >>>>
>>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay <al...@google.com> wrote:
>>> >> >>>>> >>>>>
>>> >> >>>>> >>>>> To clarify my proposal, I am proposing publishing to the production pypi repository with an rc tag in the version. And in turn allow users to depend on beam's rc version + all the other regular dependencies users would have directly from pypi.
>>> >> >>>>> >>>>>
>>> >> >>>>> >>>>> Publishing to test pypi repo would also be helpful if test pypi repo also mirrors other packages that exist in the production pypi repository.
>>> >> >>>>> >>>>>
>>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada <pa...@google.com> wrote:
>>> >> >>>>> >>>>>>
>>> >> >>>>> >>>>>> I think this is a great idea. A way of doing it for python would be by using the test repository for PyPi[1], and that way we would not have to do an official PyPi release, but still would be able to install it with pip (by passing an extra flag), and test.
>>> >> >>>>> >>>>>>
>>> >> >>>>> >>>>>> In fact, there are some Beam artifacts already in there[2]. At some point I looked into this, but couldn't figure out who has access/the password for it.
>>> >> >>>>> >>>>>
>>> >> >>>>> >>>>>
>>> >> >>>>> >>>>> I also don't know who owns beam package in test pypi repo. Does anybody know?
>>> >> >>>>> >>>>>
>>> >> >>>>> >>>>>>
>>> >> >>>>> >>>>>>
>>> >> >>>>> >>>>>> In short: +1, and I would suggest using the test PyPi repo to avoid publishing to the main PyPi repo.
>>> >> >>>>> >>>>>> Best
>>> >> >>>>> >>>>>> -P.
>>> >> >>>>> >>>>>>
>>> >> >>>>> >>>>>> [1] https://test.pypi.org/
>>> >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/
>>> >> >>>>> >>>>>>
>>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay <al...@google.com> wrote:
>>> >> >>>>> >>>>>>>
>>> >> >>>>> >>>>>>> Hi all,
>>> >> >>>>> >>>>>>>
>>> >> >>>>> >>>>>>> What do you think about the idea of publishing pre-release artifacts as part of the RC emails?
>>> >> >>>>> >>>>>>>
>>> >> >>>>> >>>>>>> For Python this would translate into publishing the same artifacts from RC email with a version like "2.X.0rcY" to pypi. I do not know, but I am guessing we can do a similar thing with Maven central for Java artifacts as well.
>>> >> >>>>> >>>>>>>
>>> >> >>>>> >>>>>>> Advantages would be:
>>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their own purposes using the same exact process they will normally use.
>>> >> >>>>> >>>>>>>   - Enable early-adaptors to start using RC releases early on in the release cycle if that is what they would like to do. This will in turn reduce time pressure on some releases. Especially for cases like someone needs a release to be finalized for an upcoming event.
>>> >> >>>>> >>>>>>>
>>> >> >>>>> >>>>>>> There will also be disadvantages, some I could think of:
>>> >> >>>>> >>>>>>> - Users could request support for RC artifacts. Hopefully in the form of feedback for us to improve the release. But it could also be in the form of folks using RC artifacts for production for a long time.
>>> >> >>>>> >>>>>>> - It will add toil to the current release process, there will be one more step for each RC. I think for python this will be a small step but nevertheless it will be additional work.
>>> >> >>>>> >>>>>>>
>>> >> >>>>> >>>>>>> For an example of this, you can take a look at tensorflow releases. For 1.13 there were 3 pre-releases [1].
>>> >> >>>>> >>>>>>>
>>> >> >>>>> >>>>>>> Ahmet
>>> >> >>>>> >>>>>>>
>>> >> >>>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history