You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Stamatis Zampetakis <za...@gmail.com> on 2025/10/22 08:19:14 UTC
[DISCUSS] Hive 4.2.0 release
Hey team,
It's already been ~3 months since our last 4.1.0 release so if we want
to stick to a quarterly release schedule we will have to cut the next
release candidate in the next 1-2 weeks.
1. How do you feel about this timeline ?
2. Who can serve as the release manager for 4.2.0 ?
3. Is the current master in a releasable state (no regressions, no big
must have patches)?
Looking forward to your thoughts.
Best,
Stamatis
Re: [DISCUSS] Hive 4.2.0 release
Posted by Dmitriy Fingerman <dm...@gmail.com>.
Hey team,
I intend to cut the branch for the 4.2.0 release in a few days.
In the meanwhile, I'm preparing a list of major open issues that I'll share
here to discuss if they need to be fixed for the 4.2.0 release.
Thanks,
Dmitriy
On Mon, Oct 27, 2025 at 11:51 AM Dmitriy Fingerman <
dmitriy.fingerman.oss@gmail.com> wrote:
> Hi All,
>
> I don't have any concerns about the release timeline and I think we have
> enough features for the release.
> I’d like to volunteer to serve as the release manager for this release.
>
> Thanks,
> Dmitriy Fingerman
>
> On Mon, Oct 27, 2025 at 7:58 AM Denys Kuzmenko <dk...@apache.org>
> wrote:
>
>>
>> > I don't think we have the necessary (human & CI) resources to maintain
>> multiple release lines so from my perspective when 4.2.0 goes out 4.1.0
>> should go automatically EOL;
>>
>> Fully agree
>>
>> > Regarding HIVE-29145 and HIVE-29213, I posted some thoughts under the
>> respective tickets and I would encourage others to do that as well.
>>
>> The filter below returns additional tickets with 4.1.0 listed as the
>> affected version. We should consider reviewing those as well:
>>
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20and%20affectedVersion%20%3D%204.1.0%20and%20type%20%3D%20Bug%20and%20resolution%20%3D%20Unresolved%20order%20BY%20created%20DESC
>>
>> Regarding the Spark issue. Frankly, I’m not too concerned about their
>> design choice to include Hive 2.3 jars in Spark 3.5/4.0 while aiming for
>> HMS 4.0 compatibility. Here’s some further context:
>> https://github.com/apache/iceberg/issues/12878
>>
>> Regards,
>> Denys
>>
>
Re: [DISCUSS] Hive 4.2.0 release
Posted by Dmitriy Fingerman <dm...@gmail.com>.
Hi All,
I don't have any concerns about the release timeline and I think we have
enough features for the release.
I’d like to volunteer to serve as the release manager for this release.
Thanks,
Dmitriy Fingerman
On Mon, Oct 27, 2025 at 7:58 AM Denys Kuzmenko <dk...@apache.org> wrote:
>
> > I don't think we have the necessary (human & CI) resources to maintain
> multiple release lines so from my perspective when 4.2.0 goes out 4.1.0
> should go automatically EOL;
>
> Fully agree
>
> > Regarding HIVE-29145 and HIVE-29213, I posted some thoughts under the
> respective tickets and I would encourage others to do that as well.
>
> The filter below returns additional tickets with 4.1.0 listed as the
> affected version. We should consider reviewing those as well:
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20and%20affectedVersion%20%3D%204.1.0%20and%20type%20%3D%20Bug%20and%20resolution%20%3D%20Unresolved%20order%20BY%20created%20DESC
>
> Regarding the Spark issue. Frankly, I’m not too concerned about their
> design choice to include Hive 2.3 jars in Spark 3.5/4.0 while aiming for
> HMS 4.0 compatibility. Here’s some further context:
> https://github.com/apache/iceberg/issues/12878
>
> Regards,
> Denys
>
Re: [DISCUSS] Hive 4.2.0 release
Posted by Denys Kuzmenko <dk...@apache.org>.
> I don't think we have the necessary (human & CI) resources to maintain multiple release lines so from my perspective when 4.2.0 goes out 4.1.0 should go automatically EOL;
Fully agree
> Regarding HIVE-29145 and HIVE-29213, I posted some thoughts under the respective tickets and I would encourage others to do that as well.
The filter below returns additional tickets with 4.1.0 listed as the affected version. We should consider reviewing those as well:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20and%20affectedVersion%20%3D%204.1.0%20and%20type%20%3D%20Bug%20and%20resolution%20%3D%20Unresolved%20order%20BY%20created%20DESC
Regarding the Spark issue. Frankly, I’m not too concerned about their design choice to include Hive 2.3 jars in Spark 3.5/4.0 while aiming for HMS 4.0 compatibility. Here’s some further context:
https://github.com/apache/iceberg/issues/12878
Regards,
Denys
Re: [DISCUSS] Hive 4.2.0 release
Posted by Stamatis Zampetakis <za...@gmail.com>.
As I advocated in previous threads I don't think we have the necessary
(human & CI) resources to maintain multiple release lines so from my
perspective when 4.2.0 goes out 4.1.0 should go automatically EOL; in
other words there won't be a 4.1.1 release. We can continue iterating
on this discussion but it's important to converge before we publish
the 4.2.0 so that there is a clear communication to our users.
Regarding HIVE-29145 and HIVE-29213, I posted some thoughts under the
respective tickets and I would encourage others to do that as well.
Regarding the addition of the get_table method, I don't fully
understand how it would help Spark if they don't upgrade to a
supported version but feel free to log a jira ticket and we can
iterate there to find a solution.
Deprecation and removal is a very important discussion but slightly
off scope in the current thread. Both contributors and reviewers
should carefully evaluate these aspects when raising/accepting PRs.
Any concrete ideas/initiatives to improve on this aspect are always
welcome so feel free to start/submit threads/docs/tickets/tests on
this topic.
Best,
Stamatis
On Sat, Oct 25, 2025 at 5:55 AM Butao Zhang <zh...@apache.org> wrote:
>
> Thank you, Okumin, for taking on the responsibility of the next Hive release.
>
> I'd also like to discuss an issue here. Since Hive 4.0.1, Apache Spark has been unable to query Iceberg tables in HMS. The specific issue discussion can be found at [1]. This problem occurred because we removed the `get_table` method in [2]. Spark relies on Hive 2.3.10, which depends on the `get_table` method to access Iceberg tables in HMS. The compatibility issue emerged after HIVE-26537.
>
> To resolve this, the ideal solution would be for Apache Spark to upgrade its built-in Hive dependency from version 2.3.10 to 4.0.1 or 4.1.0. I've noticed that there is an upgrade ticket [3] in the Spark community for this. However, this upgrade appears to involve significant changes and may not be resolved for quite some time.
>
> Given that Apache Spark is one of the most widely used engines with HMS and its adoption would help promote the use of Hive 4, is it possible for us to reintroduce the removed `get_table` method? I understand that restoring deprecated code is generally not a reasonable approach, so I'd like to ask if there are any more elegant solutions on the HMS side to address this issue?
>
> Additionally, I have a personal thought: when removing or deprecating public methods in Hive, we should carefully consider the dependencies of downstream components on these methods to avoid long-term upgrade difficulties for them. For example, when working on federated catalog [4] related tasks, I needed to add catalog parameters to many methods in `Hive.java`. However, since Spark code depends on `Hive.java` to access Hive, I had to introduce many new methods with catalog parameters, which may lead to significant code duplication. But there was no alternative, as modifying existing methods would cause compatibility issues for downstream components with newer Hive versions. Currently, this incompatibility risk relies on individual prior knowledge, but I hope we can develop better methods in the future to proactively identify such compatibility issues.
>
> [1]https://github.com/apache/iceberg/issues/12878
> [2]https://issues.apache.org/jira/browse/HIVE-26537
> [3]https://issues.apache.org/jira/browse/SPARK-52408
> [4]https://issues.apache.org/jira/browse/HIVE-29177
>
> Thanks,
> Butao Zhang
>
> On 2025/10/24 09:03:28 Shohei Okumiya wrote:
> > Hi Stamatis,
> >
> > Thanks for initiating this discussion. I'd love to have the great
> > release cadence!
> >
> > 1. I don't have any objections about the timeline. We might want to
> > release 4.1.1 in the future. I feel it should not block us from
> > shipping the master branch anyway.
> > 2. I'm willing to try being a release manager someday, but I have
> > frequent hospital visits nowadays. I'm probably not the right person,
> > given my unstable availability.
> > 3. Except for issues that have already been discussed in mailing
> > lists, I am not aware of critical problems.
> >
> > Best,
> > Okumin
> >
> > On Thu, Oct 23, 2025 at 11:50 AM Butao Zhang <zh...@apache.org> wrote:
> > >
> > > Should we release version 4.1.1? If we decide to proceed with 4.1.1, we may need to identify valuable patches from recent commits for the 4.1 branch, such as fixes for correctness issues, performance optimizations, and security problems.
> > >
> > > If we skip 4.1.1 and instead release 4.2.0, would this imply that we've reached a default agreement to only publish major version releases based on the master branch going forward?
> > >
> > > Additionally, I have two tickets that may need to be addressed in the next new version: one is HIVE-29145 (Iceberg Rest server cannot work if launched by non-standalone HMS), and the other is HIVE-29213 (HS2 reports 'Failed to get primary keys' when using old beeline client).
> > >
> > > Thanks,
> > > Butao Zhang
> > >
> > >
> > > On 2025/10/22 08:48:19 Denys Kuzmenko wrote:
> > > > Hi Stamatis,
> > > >
> > > > Thanks for bringing this up.
> > > >
> > > > We already have enough items ready to ship, including JDK 21.
> > > > I'm not sure what’s causing the delay. I thought we already had a release manager, but looks like we don’t.
> > > >
> > > > Regards,
> > > > Denys
> > > >
> >
Re: [DISCUSS] Hive 4.2.0 release
Posted by Butao Zhang <zh...@apache.org>.
Thank you, Okumin, for taking on the responsibility of the next Hive release.
I'd also like to discuss an issue here. Since Hive 4.0.1, Apache Spark has been unable to query Iceberg tables in HMS. The specific issue discussion can be found at [1]. This problem occurred because we removed the `get_table` method in [2]. Spark relies on Hive 2.3.10, which depends on the `get_table` method to access Iceberg tables in HMS. The compatibility issue emerged after HIVE-26537.
To resolve this, the ideal solution would be for Apache Spark to upgrade its built-in Hive dependency from version 2.3.10 to 4.0.1 or 4.1.0. I've noticed that there is an upgrade ticket [3] in the Spark community for this. However, this upgrade appears to involve significant changes and may not be resolved for quite some time.
Given that Apache Spark is one of the most widely used engines with HMS and its adoption would help promote the use of Hive 4, is it possible for us to reintroduce the removed `get_table` method? I understand that restoring deprecated code is generally not a reasonable approach, so I'd like to ask if there are any more elegant solutions on the HMS side to address this issue?
Additionally, I have a personal thought: when removing or deprecating public methods in Hive, we should carefully consider the dependencies of downstream components on these methods to avoid long-term upgrade difficulties for them. For example, when working on federated catalog [4] related tasks, I needed to add catalog parameters to many methods in `Hive.java`. However, since Spark code depends on `Hive.java` to access Hive, I had to introduce many new methods with catalog parameters, which may lead to significant code duplication. But there was no alternative, as modifying existing methods would cause compatibility issues for downstream components with newer Hive versions. Currently, this incompatibility risk relies on individual prior knowledge, but I hope we can develop better methods in the future to proactively identify such compatibility issues.
[1]https://github.com/apache/iceberg/issues/12878
[2]https://issues.apache.org/jira/browse/HIVE-26537
[3]https://issues.apache.org/jira/browse/SPARK-52408
[4]https://issues.apache.org/jira/browse/HIVE-29177
Thanks,
Butao Zhang
On 2025/10/24 09:03:28 Shohei Okumiya wrote:
> Hi Stamatis,
>
> Thanks for initiating this discussion. I'd love to have the great
> release cadence!
>
> 1. I don't have any objections about the timeline. We might want to
> release 4.1.1 in the future. I feel it should not block us from
> shipping the master branch anyway.
> 2. I'm willing to try being a release manager someday, but I have
> frequent hospital visits nowadays. I'm probably not the right person,
> given my unstable availability.
> 3. Except for issues that have already been discussed in mailing
> lists, I am not aware of critical problems.
>
> Best,
> Okumin
>
> On Thu, Oct 23, 2025 at 11:50 AM Butao Zhang <zh...@apache.org> wrote:
> >
> > Should we release version 4.1.1? If we decide to proceed with 4.1.1, we may need to identify valuable patches from recent commits for the 4.1 branch, such as fixes for correctness issues, performance optimizations, and security problems.
> >
> > If we skip 4.1.1 and instead release 4.2.0, would this imply that we've reached a default agreement to only publish major version releases based on the master branch going forward?
> >
> > Additionally, I have two tickets that may need to be addressed in the next new version: one is HIVE-29145 (Iceberg Rest server cannot work if launched by non-standalone HMS), and the other is HIVE-29213 (HS2 reports 'Failed to get primary keys' when using old beeline client).
> >
> > Thanks,
> > Butao Zhang
> >
> >
> > On 2025/10/22 08:48:19 Denys Kuzmenko wrote:
> > > Hi Stamatis,
> > >
> > > Thanks for bringing this up.
> > >
> > > We already have enough items ready to ship, including JDK 21.
> > > I'm not sure what’s causing the delay. I thought we already had a release manager, but looks like we don’t.
> > >
> > > Regards,
> > > Denys
> > >
>
Re: [DISCUSS] Hive 4.2.0 release
Posted by Shohei Okumiya <ok...@apache.org>.
Hi Stamatis,
Thanks for initiating this discussion. I'd love to have the great
release cadence!
1. I don't have any objections about the timeline. We might want to
release 4.1.1 in the future. I feel it should not block us from
shipping the master branch anyway.
2. I'm willing to try being a release manager someday, but I have
frequent hospital visits nowadays. I'm probably not the right person,
given my unstable availability.
3. Except for issues that have already been discussed in mailing
lists, I am not aware of critical problems.
Best,
Okumin
On Thu, Oct 23, 2025 at 11:50 AM Butao Zhang <zh...@apache.org> wrote:
>
> Should we release version 4.1.1? If we decide to proceed with 4.1.1, we may need to identify valuable patches from recent commits for the 4.1 branch, such as fixes for correctness issues, performance optimizations, and security problems.
>
> If we skip 4.1.1 and instead release 4.2.0, would this imply that we've reached a default agreement to only publish major version releases based on the master branch going forward?
>
> Additionally, I have two tickets that may need to be addressed in the next new version: one is HIVE-29145 (Iceberg Rest server cannot work if launched by non-standalone HMS), and the other is HIVE-29213 (HS2 reports 'Failed to get primary keys' when using old beeline client).
>
> Thanks,
> Butao Zhang
>
>
> On 2025/10/22 08:48:19 Denys Kuzmenko wrote:
> > Hi Stamatis,
> >
> > Thanks for bringing this up.
> >
> > We already have enough items ready to ship, including JDK 21.
> > I'm not sure what’s causing the delay. I thought we already had a release manager, but looks like we don’t.
> >
> > Regards,
> > Denys
> >
Re: [DISCUSS] Hive 4.2.0 release
Posted by Butao Zhang <zh...@apache.org>.
Should we release version 4.1.1? If we decide to proceed with 4.1.1, we may need to identify valuable patches from recent commits for the 4.1 branch, such as fixes for correctness issues, performance optimizations, and security problems.
If we skip 4.1.1 and instead release 4.2.0, would this imply that we've reached a default agreement to only publish major version releases based on the master branch going forward?
Additionally, I have two tickets that may need to be addressed in the next new version: one is HIVE-29145 (Iceberg Rest server cannot work if launched by non-standalone HMS), and the other is HIVE-29213 (HS2 reports 'Failed to get primary keys' when using old beeline client).
Thanks,
Butao Zhang
On 2025/10/22 08:48:19 Denys Kuzmenko wrote:
> Hi Stamatis,
>
> Thanks for bringing this up.
>
> We already have enough items ready to ship, including JDK 21.
> I'm not sure what’s causing the delay. I thought we already had a release manager, but looks like we don’t.
>
> Regards,
> Denys
>
Re: [DISCUSS] Hive 4.2.0 release
Posted by Denys Kuzmenko <dk...@apache.org>.
Hi Stamatis,
Thanks for bringing this up.
We already have enough items ready to ship, including JDK 21.
I'm not sure what’s causing the delay. I thought we already had a release manager, but looks like we don’t.
Regards,
Denys