You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Hongyu Guo <gu...@gmail.com> on 2023/11/04 13:03:25 UTC
How about enable dependabot?
Hi all,
Recently, I opened 2 PRs about removing an unused library[1] and bumping
various libraries[2]. I noticed that many dependencies of calcite are
outdated. To address this issue, I suggest enabling dependabot[3] to
automatically open "bump dependency" PRs and make calcite healthier.
If we enable dependabot, what should we do?
- Add `dependabot.yml` to `.github/`. It is straightforward, just follow
the instructions in the documentation[4].
- Refactor gradle project files: Dependabot's support for gradle is not
sufficient as it only reads the text of `build.gradle.kts`, `build.gradle`,
and `settings.gradle.kts` instead of running gradle. Additionally,
dependabot can NOT read `gradle.properties`, so we need to refactor the
gradle project files.
- Ignore some dependencies: Some dependencies cannot be upgraded. For
example, I attempted to bump javacc from 4.0 to 7.x, but due to
incompatibility caused by the large version span, I had to give up. Also,
we cannot upgrade elasticsearch due to licensing restrictions.
What is your opinion on dependabot?
[1]https://github.com/apache/calcite/pull/3502
[2]https://github.com/apache/calcite/pull/3504
[3]
https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/about-dependabot-version-updates
[4]
https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#gradle
Best,
Hongyu
Re: How about enable dependabot?
Posted by Guillaume Masse <ma...@narrative.io.INVALID>.
To remove some of the friction, we could setup schedule to monthly:
https://docs.github.com/fr/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#scheduleinterval
this would remove the noise, but have a checkpoint where we can bump
dependencies
On Mon, Nov 6, 2023 at 3:27 AM Stamatis Zampetakis <za...@gmail.com>
wrote:
> Upgrading dependencies is an important topic thanks for starting the
> discussion Hongyu.
>
> In terms of tooling, my experience with Dependabot in Apache Hive is
> rather negative. Out of 52 PRs [1] raised by the bot 34 [2] are
> failing the build/tests. In most cases committers do not follow-up to
> resolve the failures and currently there are only 3 [3] that are
> merged to master. In Hive, the presence of the bot adds noise and
> wastes resources.
>
> Calcite is a simpler project so maybe Dependabot does a better job but
> unless there are people who are willing to track and follow up on
> those PRs the result may be similar.
>
> Best,
> Stamatis
>
> [1]
> https://github.com/apache/hive/pulls?q=is%3Apr+author%3Aapp%2Fdependabot+
> [2]
> https://github.com/apache/hive/pulls?q=is%3Apr+author%3Aapp%2Fdependabot+label%3A%22tests+unstable%22%2C%22tests+failed%22
> [3] https://github.com/apache/hive/commits?author=dependabot%5Bbot%5D
>
> On Sun, Nov 5, 2023 at 10:03 PM Francis Chuang <fr...@apache.org>
> wrote:
> >
> > Perhaps refreshVersions [1] can be used.
> >
> > [1] https://splitties.github.io/refreshVersions/
> >
> > On 6/11/2023 7:54 am, Julian Hyde wrote:
> > > I agree that we should be trying to stay on the most recent version of
> > > our dependencies (with a few exceptions, such as JavaCC). Most of our
> > > dependencies are mature libraries, and the latest version is more
> > > likely to fix security problems than to introduce bugs.
> > >
> > > However, I'm not sure that Dependabot is the best way to do it. One,
> > > dependabot generates quite a lot of noise (frequent upgrades). Two, we
> > > would have to restructure our build files. The best process is
> > > probably to do manual upgrades, like
> > > https://github.com/apache/calcite/pull/3504, just before each release.
> > > Is there a straightforward way to script those upgrades?
> > >
> > > Julian
> > >
> > >
> > > On Sun, Nov 5, 2023 at 1:09 AM Jiajun Xie <ji...@gmail.com>
> wrote:
> > >>
> > >> Hi, Hongyu.
> > >>
> > >> Your idea is great and you also introduced the steps to use it.
> > >>
> > >> We need more feedback about benefits and risks from calcite users.
> > >> # What are the benefits?
> > >> - Quickly fix dependency vulnerabilities.
> > >> - Balancing the workload of each upgrade(Not 4.0 to 7.x).
> > >> - ...
> > >>
> > >> # What are the risks?
> > >> - The latest version may be unstable.
> > >> - The burden of upgrading Calcite for users has increased.
> > >> - ...
> > >>
> > >> For me, the risks are acceptable.
> > >> I am willing to help you complete this work.
> > >>
> > >> On Sat, 4 Nov 2023 at 21:04, Hongyu Guo <gu...@gmail.com>
> wrote:
> > >>
> > >>> Hi all,
> > >>>
> > >>> Recently, I opened 2 PRs about removing an unused library[1] and
> bumping
> > >>> various libraries[2]. I noticed that many dependencies of calcite are
> > >>> outdated. To address this issue, I suggest enabling dependabot[3] to
> > >>> automatically open "bump dependency" PRs and make calcite healthier.
> > >>>
> > >>> If we enable dependabot, what should we do?
> > >>>
> > >>> - Add `dependabot.yml` to `.github/`. It is straightforward, just
> follow
> > >>> the instructions in the documentation[4].
> > >>> - Refactor gradle project files: Dependabot's support for gradle is
> not
> > >>> sufficient as it only reads the text of `build.gradle.kts`,
> `build.gradle`,
> > >>> and `settings.gradle.kts` instead of running gradle. Additionally,
> > >>> dependabot can NOT read `gradle.properties`, so we need to refactor
> the
> > >>> gradle project files.
> > >>> - Ignore some dependencies: Some dependencies cannot be upgraded. For
> > >>> example, I attempted to bump javacc from 4.0 to 7.x, but due to
> > >>> incompatibility caused by the large version span, I had to give up.
> Also,
> > >>> we cannot upgrade elasticsearch due to licensing restrictions.
> > >>>
> > >>> What is your opinion on dependabot?
> > >>>
> > >>> [1]https://github.com/apache/calcite/pull/3502
> > >>> [2]https://github.com/apache/calcite/pull/3504
> > >>> [3]
> > >>>
> > >>>
> https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/about-dependabot-version-updates
> > >>> [4]
> > >>>
> > >>>
> https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#gradle
> > >>>
> > >>> Best,
> > >>> Hongyu
> > >>>
>
Re: How about enable dependabot?
Posted by Stamatis Zampetakis <za...@gmail.com>.
Upgrading dependencies is an important topic thanks for starting the
discussion Hongyu.
In terms of tooling, my experience with Dependabot in Apache Hive is
rather negative. Out of 52 PRs [1] raised by the bot 34 [2] are
failing the build/tests. In most cases committers do not follow-up to
resolve the failures and currently there are only 3 [3] that are
merged to master. In Hive, the presence of the bot adds noise and
wastes resources.
Calcite is a simpler project so maybe Dependabot does a better job but
unless there are people who are willing to track and follow up on
those PRs the result may be similar.
Best,
Stamatis
[1] https://github.com/apache/hive/pulls?q=is%3Apr+author%3Aapp%2Fdependabot+
[2] https://github.com/apache/hive/pulls?q=is%3Apr+author%3Aapp%2Fdependabot+label%3A%22tests+unstable%22%2C%22tests+failed%22
[3] https://github.com/apache/hive/commits?author=dependabot%5Bbot%5D
On Sun, Nov 5, 2023 at 10:03 PM Francis Chuang <fr...@apache.org> wrote:
>
> Perhaps refreshVersions [1] can be used.
>
> [1] https://splitties.github.io/refreshVersions/
>
> On 6/11/2023 7:54 am, Julian Hyde wrote:
> > I agree that we should be trying to stay on the most recent version of
> > our dependencies (with a few exceptions, such as JavaCC). Most of our
> > dependencies are mature libraries, and the latest version is more
> > likely to fix security problems than to introduce bugs.
> >
> > However, I'm not sure that Dependabot is the best way to do it. One,
> > dependabot generates quite a lot of noise (frequent upgrades). Two, we
> > would have to restructure our build files. The best process is
> > probably to do manual upgrades, like
> > https://github.com/apache/calcite/pull/3504, just before each release.
> > Is there a straightforward way to script those upgrades?
> >
> > Julian
> >
> >
> > On Sun, Nov 5, 2023 at 1:09 AM Jiajun Xie <ji...@gmail.com> wrote:
> >>
> >> Hi, Hongyu.
> >>
> >> Your idea is great and you also introduced the steps to use it.
> >>
> >> We need more feedback about benefits and risks from calcite users.
> >> # What are the benefits?
> >> - Quickly fix dependency vulnerabilities.
> >> - Balancing the workload of each upgrade(Not 4.0 to 7.x).
> >> - ...
> >>
> >> # What are the risks?
> >> - The latest version may be unstable.
> >> - The burden of upgrading Calcite for users has increased.
> >> - ...
> >>
> >> For me, the risks are acceptable.
> >> I am willing to help you complete this work.
> >>
> >> On Sat, 4 Nov 2023 at 21:04, Hongyu Guo <gu...@gmail.com> wrote:
> >>
> >>> Hi all,
> >>>
> >>> Recently, I opened 2 PRs about removing an unused library[1] and bumping
> >>> various libraries[2]. I noticed that many dependencies of calcite are
> >>> outdated. To address this issue, I suggest enabling dependabot[3] to
> >>> automatically open "bump dependency" PRs and make calcite healthier.
> >>>
> >>> If we enable dependabot, what should we do?
> >>>
> >>> - Add `dependabot.yml` to `.github/`. It is straightforward, just follow
> >>> the instructions in the documentation[4].
> >>> - Refactor gradle project files: Dependabot's support for gradle is not
> >>> sufficient as it only reads the text of `build.gradle.kts`, `build.gradle`,
> >>> and `settings.gradle.kts` instead of running gradle. Additionally,
> >>> dependabot can NOT read `gradle.properties`, so we need to refactor the
> >>> gradle project files.
> >>> - Ignore some dependencies: Some dependencies cannot be upgraded. For
> >>> example, I attempted to bump javacc from 4.0 to 7.x, but due to
> >>> incompatibility caused by the large version span, I had to give up. Also,
> >>> we cannot upgrade elasticsearch due to licensing restrictions.
> >>>
> >>> What is your opinion on dependabot?
> >>>
> >>> [1]https://github.com/apache/calcite/pull/3502
> >>> [2]https://github.com/apache/calcite/pull/3504
> >>> [3]
> >>>
> >>> https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/about-dependabot-version-updates
> >>> [4]
> >>>
> >>> https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#gradle
> >>>
> >>> Best,
> >>> Hongyu
> >>>
Re: How about enable dependabot?
Posted by Francis Chuang <fr...@apache.org>.
Perhaps refreshVersions [1] can be used.
[1] https://splitties.github.io/refreshVersions/
On 6/11/2023 7:54 am, Julian Hyde wrote:
> I agree that we should be trying to stay on the most recent version of
> our dependencies (with a few exceptions, such as JavaCC). Most of our
> dependencies are mature libraries, and the latest version is more
> likely to fix security problems than to introduce bugs.
>
> However, I'm not sure that Dependabot is the best way to do it. One,
> dependabot generates quite a lot of noise (frequent upgrades). Two, we
> would have to restructure our build files. The best process is
> probably to do manual upgrades, like
> https://github.com/apache/calcite/pull/3504, just before each release.
> Is there a straightforward way to script those upgrades?
>
> Julian
>
>
> On Sun, Nov 5, 2023 at 1:09 AM Jiajun Xie <ji...@gmail.com> wrote:
>>
>> Hi, Hongyu.
>>
>> Your idea is great and you also introduced the steps to use it.
>>
>> We need more feedback about benefits and risks from calcite users.
>> # What are the benefits?
>> - Quickly fix dependency vulnerabilities.
>> - Balancing the workload of each upgrade(Not 4.0 to 7.x).
>> - ...
>>
>> # What are the risks?
>> - The latest version may be unstable.
>> - The burden of upgrading Calcite for users has increased.
>> - ...
>>
>> For me, the risks are acceptable.
>> I am willing to help you complete this work.
>>
>> On Sat, 4 Nov 2023 at 21:04, Hongyu Guo <gu...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> Recently, I opened 2 PRs about removing an unused library[1] and bumping
>>> various libraries[2]. I noticed that many dependencies of calcite are
>>> outdated. To address this issue, I suggest enabling dependabot[3] to
>>> automatically open "bump dependency" PRs and make calcite healthier.
>>>
>>> If we enable dependabot, what should we do?
>>>
>>> - Add `dependabot.yml` to `.github/`. It is straightforward, just follow
>>> the instructions in the documentation[4].
>>> - Refactor gradle project files: Dependabot's support for gradle is not
>>> sufficient as it only reads the text of `build.gradle.kts`, `build.gradle`,
>>> and `settings.gradle.kts` instead of running gradle. Additionally,
>>> dependabot can NOT read `gradle.properties`, so we need to refactor the
>>> gradle project files.
>>> - Ignore some dependencies: Some dependencies cannot be upgraded. For
>>> example, I attempted to bump javacc from 4.0 to 7.x, but due to
>>> incompatibility caused by the large version span, I had to give up. Also,
>>> we cannot upgrade elasticsearch due to licensing restrictions.
>>>
>>> What is your opinion on dependabot?
>>>
>>> [1]https://github.com/apache/calcite/pull/3502
>>> [2]https://github.com/apache/calcite/pull/3504
>>> [3]
>>>
>>> https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/about-dependabot-version-updates
>>> [4]
>>>
>>> https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#gradle
>>>
>>> Best,
>>> Hongyu
>>>
Re: How about enable dependabot?
Posted by Julian Hyde <jh...@apache.org>.
I agree that we should be trying to stay on the most recent version of
our dependencies (with a few exceptions, such as JavaCC). Most of our
dependencies are mature libraries, and the latest version is more
likely to fix security problems than to introduce bugs.
However, I'm not sure that Dependabot is the best way to do it. One,
dependabot generates quite a lot of noise (frequent upgrades). Two, we
would have to restructure our build files. The best process is
probably to do manual upgrades, like
https://github.com/apache/calcite/pull/3504, just before each release.
Is there a straightforward way to script those upgrades?
Julian
On Sun, Nov 5, 2023 at 1:09 AM Jiajun Xie <ji...@gmail.com> wrote:
>
> Hi, Hongyu.
>
> Your idea is great and you also introduced the steps to use it.
>
> We need more feedback about benefits and risks from calcite users.
> # What are the benefits?
> - Quickly fix dependency vulnerabilities.
> - Balancing the workload of each upgrade(Not 4.0 to 7.x).
> - ...
>
> # What are the risks?
> - The latest version may be unstable.
> - The burden of upgrading Calcite for users has increased.
> - ...
>
> For me, the risks are acceptable.
> I am willing to help you complete this work.
>
> On Sat, 4 Nov 2023 at 21:04, Hongyu Guo <gu...@gmail.com> wrote:
>
> > Hi all,
> >
> > Recently, I opened 2 PRs about removing an unused library[1] and bumping
> > various libraries[2]. I noticed that many dependencies of calcite are
> > outdated. To address this issue, I suggest enabling dependabot[3] to
> > automatically open "bump dependency" PRs and make calcite healthier.
> >
> > If we enable dependabot, what should we do?
> >
> > - Add `dependabot.yml` to `.github/`. It is straightforward, just follow
> > the instructions in the documentation[4].
> > - Refactor gradle project files: Dependabot's support for gradle is not
> > sufficient as it only reads the text of `build.gradle.kts`, `build.gradle`,
> > and `settings.gradle.kts` instead of running gradle. Additionally,
> > dependabot can NOT read `gradle.properties`, so we need to refactor the
> > gradle project files.
> > - Ignore some dependencies: Some dependencies cannot be upgraded. For
> > example, I attempted to bump javacc from 4.0 to 7.x, but due to
> > incompatibility caused by the large version span, I had to give up. Also,
> > we cannot upgrade elasticsearch due to licensing restrictions.
> >
> > What is your opinion on dependabot?
> >
> > [1]https://github.com/apache/calcite/pull/3502
> > [2]https://github.com/apache/calcite/pull/3504
> > [3]
> >
> > https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/about-dependabot-version-updates
> > [4]
> >
> > https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#gradle
> >
> > Best,
> > Hongyu
> >
Re: How about enable dependabot?
Posted by Jiajun Xie <ji...@gmail.com>.
Hi, Hongyu.
Your idea is great and you also introduced the steps to use it.
We need more feedback about benefits and risks from calcite users.
# What are the benefits?
- Quickly fix dependency vulnerabilities.
- Balancing the workload of each upgrade(Not 4.0 to 7.x).
- ...
# What are the risks?
- The latest version may be unstable.
- The burden of upgrading Calcite for users has increased.
- ...
For me, the risks are acceptable.
I am willing to help you complete this work.
On Sat, 4 Nov 2023 at 21:04, Hongyu Guo <gu...@gmail.com> wrote:
> Hi all,
>
> Recently, I opened 2 PRs about removing an unused library[1] and bumping
> various libraries[2]. I noticed that many dependencies of calcite are
> outdated. To address this issue, I suggest enabling dependabot[3] to
> automatically open "bump dependency" PRs and make calcite healthier.
>
> If we enable dependabot, what should we do?
>
> - Add `dependabot.yml` to `.github/`. It is straightforward, just follow
> the instructions in the documentation[4].
> - Refactor gradle project files: Dependabot's support for gradle is not
> sufficient as it only reads the text of `build.gradle.kts`, `build.gradle`,
> and `settings.gradle.kts` instead of running gradle. Additionally,
> dependabot can NOT read `gradle.properties`, so we need to refactor the
> gradle project files.
> - Ignore some dependencies: Some dependencies cannot be upgraded. For
> example, I attempted to bump javacc from 4.0 to 7.x, but due to
> incompatibility caused by the large version span, I had to give up. Also,
> we cannot upgrade elasticsearch due to licensing restrictions.
>
> What is your opinion on dependabot?
>
> [1]https://github.com/apache/calcite/pull/3502
> [2]https://github.com/apache/calcite/pull/3504
> [3]
>
> https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/about-dependabot-version-updates
> [4]
>
> https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#gradle
>
> Best,
> Hongyu
>