You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Dominik Moritz <do...@apache.org> on 2021/11/04 16:03:55 UTC
Independent JS patch releases
Dear Arrow Devs,
We would like to ask to have independent patch releases for the Arrow JS
library. Having independent patch releases will help us resolve issues that
only become visible when people start to use the library. The JS ecosystem
is diverse and only recently has become more standardized. Therefore, it
has been difficult to get everything right the the first time, which had
led to frustrations by many users. It would really help us if we could make
patch releases to fix critical issues without waiting for other libraries
to be ready for a new release as well.
Thank you for considering the request,
Dominik for the Arrow JS devs
Re: Independent JS patch releases
Posted by Dominik Moritz <do...@apache.org>.
I only want to have independent patch releases but still keep the main
releases aligned with the other releases.
I want to make sure I have the process right. When we want to make a patch
release, do we still need to call a vote? When we actually want to make a
release, what do we do to bump the version number and call `npm publish`?
What’s the purpose of the source tarball script you linked to? Should this
be called by a CI process or a user with particular access permissions?
On Nov 14, 2021 at 16:26:15, Wes McKinney <we...@gmail.com> wrote:
> I think the idea would be to make a source tarball out of the JS
> directory — we had a script for this a few years ago from the last
> time we made independent JS releases — if you want to resurrect this
> and modify it for the current state of the project, this seems fine.
> It would also be helpful to have a modified release verification
> script to help streamline votes (like we have for the Rust releases,
> for example)
>
>
> https://github.com/apache/arrow/blob/ea0fb370e2ee0d11b2fbdb149247af3695fd394a/dev/release/js-source-release.sh
>
> On Fri, Nov 12, 2021 at 9:46 AM Dominik Moritz <do...@apache.org>
> wrote:
>
>
> Thank you, Wes.
>
>
> Could you point me to what preparations we need to make now and how we can
>
> make a patch release in the future?
>
>
> On Nov 10, 2021 at 17:34:44, Wes McKinney <we...@gmail.com> wrote:
>
>
> > I don't see a problem with making JS-only patch releases. There's some
>
> > work to facilitate the release management but if it's a source-only
>
> > release it shouldn't be _that_ difficult.
>
> >
>
> > On Fri, Nov 5, 2021 at 9:50 AM Dominik Moritz <do...@apache.org>
> wrote:
>
> >
>
> >
>
> > Hi Neal,
>
> >
>
> >
>
> > Great questions. We could potentially add an integration test for major
>
> >
>
> > bundlers. However, then we would also need a way to test these packages
> in
>
> >
>
> > different environments like browsers and that’s a lot of work that I’m
> not
>
> >
>
> > sure will be proportional to the benefit. The issue in my experience is
>
> >
>
> > that the JS ecosystem is very diverse and there are many bundlers and
>
> >
>
> > different ways to consume packages (just look at the list of
> configurations
>
> >
>
> > we generate in https://github.com/apache/arrow/tree/master/js#packaging
>
> > and
>
> >
>
> > that’s ignoring some other es variants like es2017 and separate bundles
> for
>
> >
>
> > browsers and node; or Deno in the future maybe). It’s really hard to test
>
> >
>
> > all of these things automatically.
>
> >
>
> >
>
> > I think pre-releases would go a long way to working packages when we
> make a
>
> >
>
> > major release but I suspect there will always be some use cases we did
> not
>
> >
>
> > anticipate. I made two JIRAs for pre-releases:
>
> >
>
> > https://issues.apache.org/jira/browse/ARROW-14508 and
>
> >
>
> > https://issues.apache.org/jira/browse/ARROW-14507.
>
> >
>
> >
>
> > Best wishes,
>
> >
>
> > Dominik
>
> >
>
> >
>
> > On Nov 5, 2021 at 10:00:10, Neal Richardson <neal.p.richardson@gmail.com
> >
>
> >
>
> > wrote:
>
> >
>
> >
>
> > > Not expressing an opinion on the original question, but if the problem
> is
>
> >
>
> > > "not getting everything right the first time", what can be done to
> reduce
>
> >
>
> > > the likelihood of getting things wrong? Other languages/implementations
>
> >
>
> > > have extensive CI and nightly builds, some of which test different
>
> >
>
> > > packaging scenarios and integration testing with other projects (e.g.
>
> >
>
> > > spark, dask) that we don't want to break support with. Are there
>
> > additional
>
> >
>
> > > JS builds we could add that would increase our confidence in the
> quality
>
> > of
>
> >
>
> > > our releases?
>
> >
>
> > >
>
> >
>
> > > Neal
>
> >
>
> > >
>
> >
>
> > > On Thu, Nov 4, 2021 at 12:04 PM Dominik Moritz <do...@apache.org>
>
> >
>
> > > wrote:
>
> >
>
> > >
>
> >
>
> > > Dear Arrow Devs,
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > We would like to ask to have independent patch releases for the Arrow
> JS
>
> >
>
> > >
>
> >
>
> > > library. Having independent patch releases will help us resolve issues
>
> > that
>
> >
>
> > >
>
> >
>
> > > only become visible when people start to use the library. The JS
>
> > ecosystem
>
> >
>
> > >
>
> >
>
> > > is diverse and only recently has become more standardized. Therefore,
> it
>
> >
>
> > >
>
> >
>
> > > has been difficult to get everything right the the first time, which
> had
>
> >
>
> > >
>
> >
>
> > > led to frustrations by many users. It would really help us if we could
>
> > make
>
> >
>
> > >
>
> >
>
> > > patch releases to fix critical issues without waiting for other
> libraries
>
> >
>
> > >
>
> >
>
> > > to be ready for a new release as well.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Thank you for considering the request,
>
> >
>
> > >
>
> >
>
> > > Dominik for the Arrow JS devs
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> >
>
>
Re: Independent JS patch releases
Posted by Wes McKinney <we...@gmail.com>.
I think the idea would be to make a source tarball out of the JS
directory — we had a script for this a few years ago from the last
time we made independent JS releases — if you want to resurrect this
and modify it for the current state of the project, this seems fine.
It would also be helpful to have a modified release verification
script to help streamline votes (like we have for the Rust releases,
for example)
https://github.com/apache/arrow/blob/ea0fb370e2ee0d11b2fbdb149247af3695fd394a/dev/release/js-source-release.sh
On Fri, Nov 12, 2021 at 9:46 AM Dominik Moritz <do...@apache.org> wrote:
>
> Thank you, Wes.
>
> Could you point me to what preparations we need to make now and how we can
> make a patch release in the future?
>
> On Nov 10, 2021 at 17:34:44, Wes McKinney <we...@gmail.com> wrote:
>
> > I don't see a problem with making JS-only patch releases. There's some
> > work to facilitate the release management but if it's a source-only
> > release it shouldn't be _that_ difficult.
> >
> > On Fri, Nov 5, 2021 at 9:50 AM Dominik Moritz <do...@apache.org> wrote:
> >
> >
> > Hi Neal,
> >
> >
> > Great questions. We could potentially add an integration test for major
> >
> > bundlers. However, then we would also need a way to test these packages in
> >
> > different environments like browsers and that’s a lot of work that I’m not
> >
> > sure will be proportional to the benefit. The issue in my experience is
> >
> > that the JS ecosystem is very diverse and there are many bundlers and
> >
> > different ways to consume packages (just look at the list of configurations
> >
> > we generate in https://github.com/apache/arrow/tree/master/js#packaging
> > and
> >
> > that’s ignoring some other es variants like es2017 and separate bundles for
> >
> > browsers and node; or Deno in the future maybe). It’s really hard to test
> >
> > all of these things automatically.
> >
> >
> > I think pre-releases would go a long way to working packages when we make a
> >
> > major release but I suspect there will always be some use cases we did not
> >
> > anticipate. I made two JIRAs for pre-releases:
> >
> > https://issues.apache.org/jira/browse/ARROW-14508 and
> >
> > https://issues.apache.org/jira/browse/ARROW-14507.
> >
> >
> > Best wishes,
> >
> > Dominik
> >
> >
> > On Nov 5, 2021 at 10:00:10, Neal Richardson <ne...@gmail.com>
> >
> > wrote:
> >
> >
> > > Not expressing an opinion on the original question, but if the problem is
> >
> > > "not getting everything right the first time", what can be done to reduce
> >
> > > the likelihood of getting things wrong? Other languages/implementations
> >
> > > have extensive CI and nightly builds, some of which test different
> >
> > > packaging scenarios and integration testing with other projects (e.g.
> >
> > > spark, dask) that we don't want to break support with. Are there
> > additional
> >
> > > JS builds we could add that would increase our confidence in the quality
> > of
> >
> > > our releases?
> >
> > >
> >
> > > Neal
> >
> > >
> >
> > > On Thu, Nov 4, 2021 at 12:04 PM Dominik Moritz <do...@apache.org>
> >
> > > wrote:
> >
> > >
> >
> > > Dear Arrow Devs,
> >
> > >
> >
> > >
> >
> > > We would like to ask to have independent patch releases for the Arrow JS
> >
> > >
> >
> > > library. Having independent patch releases will help us resolve issues
> > that
> >
> > >
> >
> > > only become visible when people start to use the library. The JS
> > ecosystem
> >
> > >
> >
> > > is diverse and only recently has become more standardized. Therefore, it
> >
> > >
> >
> > > has been difficult to get everything right the the first time, which had
> >
> > >
> >
> > > led to frustrations by many users. It would really help us if we could
> > make
> >
> > >
> >
> > > patch releases to fix critical issues without waiting for other libraries
> >
> > >
> >
> > > to be ready for a new release as well.
> >
> > >
> >
> > >
> >
> > > Thank you for considering the request,
> >
> > >
> >
> > > Dominik for the Arrow JS devs
> >
> > >
> >
> > >
> >
> > >
> >
> >
Re: Independent JS patch releases
Posted by Dominik Moritz <do...@apache.org>.
Thank you, Wes.
Could you point me to what preparations we need to make now and how we can
make a patch release in the future?
On Nov 10, 2021 at 17:34:44, Wes McKinney <we...@gmail.com> wrote:
> I don't see a problem with making JS-only patch releases. There's some
> work to facilitate the release management but if it's a source-only
> release it shouldn't be _that_ difficult.
>
> On Fri, Nov 5, 2021 at 9:50 AM Dominik Moritz <do...@apache.org> wrote:
>
>
> Hi Neal,
>
>
> Great questions. We could potentially add an integration test for major
>
> bundlers. However, then we would also need a way to test these packages in
>
> different environments like browsers and that’s a lot of work that I’m not
>
> sure will be proportional to the benefit. The issue in my experience is
>
> that the JS ecosystem is very diverse and there are many bundlers and
>
> different ways to consume packages (just look at the list of configurations
>
> we generate in https://github.com/apache/arrow/tree/master/js#packaging
> and
>
> that’s ignoring some other es variants like es2017 and separate bundles for
>
> browsers and node; or Deno in the future maybe). It’s really hard to test
>
> all of these things automatically.
>
>
> I think pre-releases would go a long way to working packages when we make a
>
> major release but I suspect there will always be some use cases we did not
>
> anticipate. I made two JIRAs for pre-releases:
>
> https://issues.apache.org/jira/browse/ARROW-14508 and
>
> https://issues.apache.org/jira/browse/ARROW-14507.
>
>
> Best wishes,
>
> Dominik
>
>
> On Nov 5, 2021 at 10:00:10, Neal Richardson <ne...@gmail.com>
>
> wrote:
>
>
> > Not expressing an opinion on the original question, but if the problem is
>
> > "not getting everything right the first time", what can be done to reduce
>
> > the likelihood of getting things wrong? Other languages/implementations
>
> > have extensive CI and nightly builds, some of which test different
>
> > packaging scenarios and integration testing with other projects (e.g.
>
> > spark, dask) that we don't want to break support with. Are there
> additional
>
> > JS builds we could add that would increase our confidence in the quality
> of
>
> > our releases?
>
> >
>
> > Neal
>
> >
>
> > On Thu, Nov 4, 2021 at 12:04 PM Dominik Moritz <do...@apache.org>
>
> > wrote:
>
> >
>
> > Dear Arrow Devs,
>
> >
>
> >
>
> > We would like to ask to have independent patch releases for the Arrow JS
>
> >
>
> > library. Having independent patch releases will help us resolve issues
> that
>
> >
>
> > only become visible when people start to use the library. The JS
> ecosystem
>
> >
>
> > is diverse and only recently has become more standardized. Therefore, it
>
> >
>
> > has been difficult to get everything right the the first time, which had
>
> >
>
> > led to frustrations by many users. It would really help us if we could
> make
>
> >
>
> > patch releases to fix critical issues without waiting for other libraries
>
> >
>
> > to be ready for a new release as well.
>
> >
>
> >
>
> > Thank you for considering the request,
>
> >
>
> > Dominik for the Arrow JS devs
>
> >
>
> >
>
> >
>
>
Re: Independent JS patch releases
Posted by Wes McKinney <we...@gmail.com>.
I don't see a problem with making JS-only patch releases. There's some
work to facilitate the release management but if it's a source-only
release it shouldn't be _that_ difficult.
On Fri, Nov 5, 2021 at 9:50 AM Dominik Moritz <do...@apache.org> wrote:
>
> Hi Neal,
>
> Great questions. We could potentially add an integration test for major
> bundlers. However, then we would also need a way to test these packages in
> different environments like browsers and that’s a lot of work that I’m not
> sure will be proportional to the benefit. The issue in my experience is
> that the JS ecosystem is very diverse and there are many bundlers and
> different ways to consume packages (just look at the list of configurations
> we generate in https://github.com/apache/arrow/tree/master/js#packaging and
> that’s ignoring some other es variants like es2017 and separate bundles for
> browsers and node; or Deno in the future maybe). It’s really hard to test
> all of these things automatically.
>
> I think pre-releases would go a long way to working packages when we make a
> major release but I suspect there will always be some use cases we did not
> anticipate. I made two JIRAs for pre-releases:
> https://issues.apache.org/jira/browse/ARROW-14508 and
> https://issues.apache.org/jira/browse/ARROW-14507.
>
> Best wishes,
> Dominik
>
> On Nov 5, 2021 at 10:00:10, Neal Richardson <ne...@gmail.com>
> wrote:
>
> > Not expressing an opinion on the original question, but if the problem is
> > "not getting everything right the first time", what can be done to reduce
> > the likelihood of getting things wrong? Other languages/implementations
> > have extensive CI and nightly builds, some of which test different
> > packaging scenarios and integration testing with other projects (e.g.
> > spark, dask) that we don't want to break support with. Are there additional
> > JS builds we could add that would increase our confidence in the quality of
> > our releases?
> >
> > Neal
> >
> > On Thu, Nov 4, 2021 at 12:04 PM Dominik Moritz <do...@apache.org>
> > wrote:
> >
> > Dear Arrow Devs,
> >
> >
> > We would like to ask to have independent patch releases for the Arrow JS
> >
> > library. Having independent patch releases will help us resolve issues that
> >
> > only become visible when people start to use the library. The JS ecosystem
> >
> > is diverse and only recently has become more standardized. Therefore, it
> >
> > has been difficult to get everything right the the first time, which had
> >
> > led to frustrations by many users. It would really help us if we could make
> >
> > patch releases to fix critical issues without waiting for other libraries
> >
> > to be ready for a new release as well.
> >
> >
> > Thank you for considering the request,
> >
> > Dominik for the Arrow JS devs
> >
> >
> >
Re: Independent JS patch releases
Posted by Dominik Moritz <do...@apache.org>.
Hi Neal,
Great questions. We could potentially add an integration test for major
bundlers. However, then we would also need a way to test these packages in
different environments like browsers and that’s a lot of work that I’m not
sure will be proportional to the benefit. The issue in my experience is
that the JS ecosystem is very diverse and there are many bundlers and
different ways to consume packages (just look at the list of configurations
we generate in https://github.com/apache/arrow/tree/master/js#packaging and
that’s ignoring some other es variants like es2017 and separate bundles for
browsers and node; or Deno in the future maybe). It’s really hard to test
all of these things automatically.
I think pre-releases would go a long way to working packages when we make a
major release but I suspect there will always be some use cases we did not
anticipate. I made two JIRAs for pre-releases:
https://issues.apache.org/jira/browse/ARROW-14508 and
https://issues.apache.org/jira/browse/ARROW-14507.
Best wishes,
Dominik
On Nov 5, 2021 at 10:00:10, Neal Richardson <ne...@gmail.com>
wrote:
> Not expressing an opinion on the original question, but if the problem is
> "not getting everything right the first time", what can be done to reduce
> the likelihood of getting things wrong? Other languages/implementations
> have extensive CI and nightly builds, some of which test different
> packaging scenarios and integration testing with other projects (e.g.
> spark, dask) that we don't want to break support with. Are there additional
> JS builds we could add that would increase our confidence in the quality of
> our releases?
>
> Neal
>
> On Thu, Nov 4, 2021 at 12:04 PM Dominik Moritz <do...@apache.org>
> wrote:
>
> Dear Arrow Devs,
>
>
> We would like to ask to have independent patch releases for the Arrow JS
>
> library. Having independent patch releases will help us resolve issues that
>
> only become visible when people start to use the library. The JS ecosystem
>
> is diverse and only recently has become more standardized. Therefore, it
>
> has been difficult to get everything right the the first time, which had
>
> led to frustrations by many users. It would really help us if we could make
>
> patch releases to fix critical issues without waiting for other libraries
>
> to be ready for a new release as well.
>
>
> Thank you for considering the request,
>
> Dominik for the Arrow JS devs
>
>
>
Re: Independent JS patch releases
Posted by Neal Richardson <ne...@gmail.com>.
Not expressing an opinion on the original question, but if the problem is
"not getting everything right the first time", what can be done to reduce
the likelihood of getting things wrong? Other languages/implementations
have extensive CI and nightly builds, some of which test different
packaging scenarios and integration testing with other projects (e.g.
spark, dask) that we don't want to break support with. Are there additional
JS builds we could add that would increase our confidence in the quality of
our releases?
Neal
On Thu, Nov 4, 2021 at 12:04 PM Dominik Moritz <do...@apache.org> wrote:
> Dear Arrow Devs,
>
> We would like to ask to have independent patch releases for the Arrow JS
> library. Having independent patch releases will help us resolve issues that
> only become visible when people start to use the library. The JS ecosystem
> is diverse and only recently has become more standardized. Therefore, it
> has been difficult to get everything right the the first time, which had
> led to frustrations by many users. It would really help us if we could make
> patch releases to fix critical issues without waiting for other libraries
> to be ready for a new release as well.
>
> Thank you for considering the request,
> Dominik for the Arrow JS devs
>