You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Chris Thomas <ct...@tracer.tech> on 2023/11/21 21:59:54 UTC
Documentation of Breaking Changes
Evening folks,
I apologize if this is not the appropriate venue for this request; if
that's the case, please let me know where I should be asking:
Earlier this month Dependabot flagged a security vulnerability with PyArrow
which prompted us to do an upgrade from v10 to v14.1 of the software.
Obviously this is a lot of major versions so the upgrade was subjected to a
bunch of tests but, alas, there was a breaking change to the way PyArrow
handled time precision that slipped through the cracks.
Upon review I'm not sure how that change could possibly have been caught.
The change-logs for the package are a verbose dump of all of the PRs
included in the release. Working out which of them constitute a breaking
change and what the implications are of that change is difficult.
Is this something that could be addressed in the project?
--
Best,
Chris Thomas
Engineering Manager - Feature Team
540.808.2782
Re: Documentation of Breaking Changes
Posted by Antoine Pitrou <an...@python.org>.
Hello,
Le 21/11/2023 à 22:59, Chris Thomas a écrit :
>
> I apologize if this is not the appropriate venue for this request; if
> that's the case, please let me know where I should be asking:
>
> Earlier this month Dependabot flagged a security vulnerability with PyArrow
> which prompted us to do an upgrade from v10 to v14.1 of the software.
> Obviously this is a lot of major versions so the upgrade was subjected to a
> bunch of tests but, alas, there was a breaking change to the way PyArrow
> handled time precision that slipped through the cracks.
Can you explain which change around time precision you observed?
Regards
Antoine.
Re: Documentation of Breaking Changes
Posted by Chris Thomas <ct...@tracer.tech>.
Having spent a while doing patch and update management for critical
infrastructure, I think exposing it directly in the Change Log is the best
possible solution. I'll make sure to let the team know that we can look out
for future issues with some creative use of GitHub search but more
visibility is always welcome.
Thanks for the prompt and thoughtful response.
On Wed, Nov 22, 2023 at 6:20 AM Raúl Cumplido <ra...@gmail.com>
wrote:
> Hi Chris,
>
> As Bryce pointed out the current process is managed with the manual
> addition of the `Breaking change` label in GitHub. In general after
> the Release there is a review process to tag some of those that were
> missing.
>
> Currently you could use the GitHub issue search. For example for
> 13.0.0 a search filter like: `label:"Breaking Change" milestone:13.0.0
> ` [1].
>
> I think you make a great point, currently it is quite difficult to
> know what are the breaking changes. I discussed in the past with
> Alenka and Bryce that we could prompt on our merge script to the
> committer whether they want to tag the issue with `Breaking change`.
> I've created the following issue [2] in order to do that. Once this is
> done we could modify our Changelog script to build a section with
> Breaking changes to make it easier for users to find those.
>
> [1]
> https://github.com/apache/arrow/issues?q=label%3A%22Breaking+Change%22+milestone%3A13.0.0
> [2] https://github.com/apache/arrow/issues/38841
>
>
> El mié, 22 nov 2023 a las 3:13, Chris Thomas (<ct...@tracer.tech>)
> escribió:
> >
> > It's 9:10pm and I should probably check in with my dev team before
> tossing
> > this off but I'm trying to give them the longest Thanksgiving break I
> can.
> >
> > I'm fairly sure it was this PR:
> https://github.com/apache/arrow/pull/35656
> >
> > Long story short, we didn't know it at the time but we were depending on
> > the forced coercion to nanoseconds.
> >
> > On Tue, Nov 21, 2023 at 5:26 PM Bryce Mecum <br...@gmail.com>
> wrote:
> >
> > > Hi Chris, this is very much the place to ask a question like this and
> > > thanks for doing so.
> > >
> > > Could we get a little more information on the specific change you were
> > > affected by just so we're all on the same page? Was this the bump from
> > > Parquet 2.4 to 2.6 [1] that happened in the PyArrow 13 release [2] or
> > > something else?
> > >
> > > Currently, breaking changes are communicated in the release blog post
> > > [2] and the corresponding GitHub Issue gets a Breaking Change label
> > > [3], as documented in our development guide [4]. When you say
> > > "change-logs", are you referring to the changelogs on our release page
> > > [5]?
> > >
> > > [1] https://github.com/apache/arrow/issues/35746
> > > [2] https://arrow.apache.org/blog/2023/08/24/13.0.0-release/
> > > [3] https://github.com/apache/arrow/labels
> > > [4] https://arrow.apache.org/docs/developers/reviewing.html#labelling
> > > [5] https://arrow.apache.org/release/13.0.0.html#changelog
> > >
> > > On Tue, Nov 21, 2023 at 1:00 PM Chris Thomas <ct...@tracer.tech>
> wrote:
> > > >
> > > > Evening folks,
> > > >
> > > > I apologize if this is not the appropriate venue for this request; if
> > > > that's the case, please let me know where I should be asking:
> > > >
> > > > Earlier this month Dependabot flagged a security vulnerability with
> > > PyArrow
> > > > which prompted us to do an upgrade from v10 to v14.1 of the software.
> > > > Obviously this is a lot of major versions so the upgrade was
> subjected
> > > to a
> > > > bunch of tests but, alas, there was a breaking change to the way
> PyArrow
> > > > handled time precision that slipped through the cracks.
> > > >
> > > > Upon review I'm not sure how that change could possibly have been
> caught.
> > > > The change-logs for the package are a verbose dump of all of the PRs
> > > > included in the release. Working out which of them constitute a
> breaking
> > > > change and what the implications are of that change is difficult.
> > > >
> > > > Is this something that could be addressed in the project?
> > > >
> > > > --
> > > >
> > > > Best,
> > > >
> > > > Chris Thomas
> > > > Engineering Manager - Feature Team
> > > > 540.808.2782
> > >
> >
> >
> > --
> >
> > Best,
> >
> > Chris Thomas
> > Engineering Manager - Feature Team
> > 540.808.2782
>
--
Best,
Chris Thomas
Engineering Manager - Feature Team
540.808.2782
Re: Documentation of Breaking Changes
Posted by Raúl Cumplido <ra...@gmail.com>.
Hi Chris,
As Bryce pointed out the current process is managed with the manual
addition of the `Breaking change` label in GitHub. In general after
the Release there is a review process to tag some of those that were
missing.
Currently you could use the GitHub issue search. For example for
13.0.0 a search filter like: `label:"Breaking Change" milestone:13.0.0
` [1].
I think you make a great point, currently it is quite difficult to
know what are the breaking changes. I discussed in the past with
Alenka and Bryce that we could prompt on our merge script to the
committer whether they want to tag the issue with `Breaking change`.
I've created the following issue [2] in order to do that. Once this is
done we could modify our Changelog script to build a section with
Breaking changes to make it easier for users to find those.
[1] https://github.com/apache/arrow/issues?q=label%3A%22Breaking+Change%22+milestone%3A13.0.0
[2] https://github.com/apache/arrow/issues/38841
El mié, 22 nov 2023 a las 3:13, Chris Thomas (<ct...@tracer.tech>) escribió:
>
> It's 9:10pm and I should probably check in with my dev team before tossing
> this off but I'm trying to give them the longest Thanksgiving break I can.
>
> I'm fairly sure it was this PR: https://github.com/apache/arrow/pull/35656
>
> Long story short, we didn't know it at the time but we were depending on
> the forced coercion to nanoseconds.
>
> On Tue, Nov 21, 2023 at 5:26 PM Bryce Mecum <br...@gmail.com> wrote:
>
> > Hi Chris, this is very much the place to ask a question like this and
> > thanks for doing so.
> >
> > Could we get a little more information on the specific change you were
> > affected by just so we're all on the same page? Was this the bump from
> > Parquet 2.4 to 2.6 [1] that happened in the PyArrow 13 release [2] or
> > something else?
> >
> > Currently, breaking changes are communicated in the release blog post
> > [2] and the corresponding GitHub Issue gets a Breaking Change label
> > [3], as documented in our development guide [4]. When you say
> > "change-logs", are you referring to the changelogs on our release page
> > [5]?
> >
> > [1] https://github.com/apache/arrow/issues/35746
> > [2] https://arrow.apache.org/blog/2023/08/24/13.0.0-release/
> > [3] https://github.com/apache/arrow/labels
> > [4] https://arrow.apache.org/docs/developers/reviewing.html#labelling
> > [5] https://arrow.apache.org/release/13.0.0.html#changelog
> >
> > On Tue, Nov 21, 2023 at 1:00 PM Chris Thomas <ct...@tracer.tech> wrote:
> > >
> > > Evening folks,
> > >
> > > I apologize if this is not the appropriate venue for this request; if
> > > that's the case, please let me know where I should be asking:
> > >
> > > Earlier this month Dependabot flagged a security vulnerability with
> > PyArrow
> > > which prompted us to do an upgrade from v10 to v14.1 of the software.
> > > Obviously this is a lot of major versions so the upgrade was subjected
> > to a
> > > bunch of tests but, alas, there was a breaking change to the way PyArrow
> > > handled time precision that slipped through the cracks.
> > >
> > > Upon review I'm not sure how that change could possibly have been caught.
> > > The change-logs for the package are a verbose dump of all of the PRs
> > > included in the release. Working out which of them constitute a breaking
> > > change and what the implications are of that change is difficult.
> > >
> > > Is this something that could be addressed in the project?
> > >
> > > --
> > >
> > > Best,
> > >
> > > Chris Thomas
> > > Engineering Manager - Feature Team
> > > 540.808.2782
> >
>
>
> --
>
> Best,
>
> Chris Thomas
> Engineering Manager - Feature Team
> 540.808.2782
Re: Documentation of Breaking Changes
Posted by Chris Thomas <ct...@tracer.tech>.
It's 9:10pm and I should probably check in with my dev team before tossing
this off but I'm trying to give them the longest Thanksgiving break I can.
I'm fairly sure it was this PR: https://github.com/apache/arrow/pull/35656
Long story short, we didn't know it at the time but we were depending on
the forced coercion to nanoseconds.
On Tue, Nov 21, 2023 at 5:26 PM Bryce Mecum <br...@gmail.com> wrote:
> Hi Chris, this is very much the place to ask a question like this and
> thanks for doing so.
>
> Could we get a little more information on the specific change you were
> affected by just so we're all on the same page? Was this the bump from
> Parquet 2.4 to 2.6 [1] that happened in the PyArrow 13 release [2] or
> something else?
>
> Currently, breaking changes are communicated in the release blog post
> [2] and the corresponding GitHub Issue gets a Breaking Change label
> [3], as documented in our development guide [4]. When you say
> "change-logs", are you referring to the changelogs on our release page
> [5]?
>
> [1] https://github.com/apache/arrow/issues/35746
> [2] https://arrow.apache.org/blog/2023/08/24/13.0.0-release/
> [3] https://github.com/apache/arrow/labels
> [4] https://arrow.apache.org/docs/developers/reviewing.html#labelling
> [5] https://arrow.apache.org/release/13.0.0.html#changelog
>
> On Tue, Nov 21, 2023 at 1:00 PM Chris Thomas <ct...@tracer.tech> wrote:
> >
> > Evening folks,
> >
> > I apologize if this is not the appropriate venue for this request; if
> > that's the case, please let me know where I should be asking:
> >
> > Earlier this month Dependabot flagged a security vulnerability with
> PyArrow
> > which prompted us to do an upgrade from v10 to v14.1 of the software.
> > Obviously this is a lot of major versions so the upgrade was subjected
> to a
> > bunch of tests but, alas, there was a breaking change to the way PyArrow
> > handled time precision that slipped through the cracks.
> >
> > Upon review I'm not sure how that change could possibly have been caught.
> > The change-logs for the package are a verbose dump of all of the PRs
> > included in the release. Working out which of them constitute a breaking
> > change and what the implications are of that change is difficult.
> >
> > Is this something that could be addressed in the project?
> >
> > --
> >
> > Best,
> >
> > Chris Thomas
> > Engineering Manager - Feature Team
> > 540.808.2782
>
--
Best,
Chris Thomas
Engineering Manager - Feature Team
540.808.2782
Re: Documentation of Breaking Changes
Posted by Bryce Mecum <br...@gmail.com>.
Hi Chris, this is very much the place to ask a question like this and
thanks for doing so.
Could we get a little more information on the specific change you were
affected by just so we're all on the same page? Was this the bump from
Parquet 2.4 to 2.6 [1] that happened in the PyArrow 13 release [2] or
something else?
Currently, breaking changes are communicated in the release blog post
[2] and the corresponding GitHub Issue gets a Breaking Change label
[3], as documented in our development guide [4]. When you say
"change-logs", are you referring to the changelogs on our release page
[5]?
[1] https://github.com/apache/arrow/issues/35746
[2] https://arrow.apache.org/blog/2023/08/24/13.0.0-release/
[3] https://github.com/apache/arrow/labels
[4] https://arrow.apache.org/docs/developers/reviewing.html#labelling
[5] https://arrow.apache.org/release/13.0.0.html#changelog
On Tue, Nov 21, 2023 at 1:00 PM Chris Thomas <ct...@tracer.tech> wrote:
>
> Evening folks,
>
> I apologize if this is not the appropriate venue for this request; if
> that's the case, please let me know where I should be asking:
>
> Earlier this month Dependabot flagged a security vulnerability with PyArrow
> which prompted us to do an upgrade from v10 to v14.1 of the software.
> Obviously this is a lot of major versions so the upgrade was subjected to a
> bunch of tests but, alas, there was a breaking change to the way PyArrow
> handled time precision that slipped through the cracks.
>
> Upon review I'm not sure how that change could possibly have been caught.
> The change-logs for the package are a verbose dump of all of the PRs
> included in the release. Working out which of them constitute a breaking
> change and what the implications are of that change is difficult.
>
> Is this something that could be addressed in the project?
>
> --
>
> Best,
>
> Chris Thomas
> Engineering Manager - Feature Team
> 540.808.2782