You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Fokko Driesprong <fo...@apache.org> on 2024/02/16 09:09:50 UTC
Java Iceberg 2.0: Hadoop upgrade
Hi everyone,
I want to discuss adding the Hadoop upgrade to the list after moving to
Iceberg 2.0. We still compile against Hadoop 2.7.3 to ensure we support as
many users as possible. Hadoop 2.7.3 was released August 2016
<https://hadoop.apache.org/release/2.7.3.html> and is not maintained anymore
<https://endoflife.date/apache-hadoop> for a long time.
My main reason for doing the upgrade is that on the Parquet MR project,
I've been pushing back the Hadoop upgrade to ensure compatibility with
Iceberg. However, at some point, we have to pull the trigger here. This
will simplify things on the Parquet side and avoid having to check if the
Java API Exists and such.
Since Hadoop 3.3+ officially supports Java 11
<https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions>,
I would suggest dropping everything below that. I wanted to check on the
mailing list if there are any thoughts and or concerns.
Kind regards,
Fokko
Re: Java Iceberg 2.0: Hadoop upgrade
Posted by Renjie Liu <li...@gmail.com>.
+1 for upgrading Hadoop dependency versions, and add jdk 17 as the minimum
required version.
On Sat, Feb 17, 2024 at 9:18 AM Ajantha Bhat <aj...@gmail.com> wrote:
> +1 for upgrading Hadoop dependency versions.
>
> And I echo the same thoughts as Eduard for Iceberg's JDK 8 requirement.
> It will be good to have at least Java 11/17 as the minimum
> required version.
>
> - Ajantha
>
> On Fri, Feb 16, 2024 at 7:47 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi Fokko,
>>
>> I agree to remove Hadoop 2.7.3 and bump to 3.x for Iceberg 2.0.0.
>>
>> If we want to be based on Java 11+ (17 would be good imho), we have to
>> check the dependencies. I'm not sure all of our dependencies support
>> JDK17 today.
>>
>> Regards
>> JB
>>
>> On Fri, Feb 16, 2024 at 10:09 AM Fokko Driesprong <fo...@apache.org>
>> wrote:
>> >
>> > Hi everyone,
>> >
>> > I want to discuss adding the Hadoop upgrade to the list after moving to
>> Iceberg 2.0. We still compile against Hadoop 2.7.3 to ensure we support as
>> many users as possible. Hadoop 2.7.3 was released August 2016 and is not
>> maintained anymore for a long time.
>> >
>> > My main reason for doing the upgrade is that on the Parquet MR project,
>> I've been pushing back the Hadoop upgrade to ensure compatibility with
>> Iceberg. However, at some point, we have to pull the trigger here. This
>> will simplify things on the Parquet side and avoid having to check if the
>> Java API Exists and such.
>> >
>> > Since Hadoop 3.3+ officially supports Java 11, I would suggest dropping
>> everything below that. I wanted to check on the mailing list if there are
>> any thoughts and or concerns.
>> >
>> > Kind regards,
>> > Fokko
>> >
>> >
>>
>
Re: Java Iceberg 2.0: Hadoop upgrade
Posted by Ajantha Bhat <aj...@gmail.com>.
+1 for upgrading Hadoop dependency versions.
And I echo the same thoughts as Eduard for Iceberg's JDK 8 requirement.
It will be good to have at least Java 11/17 as the minimum
required version.
- Ajantha
On Fri, Feb 16, 2024 at 7:47 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Fokko,
>
> I agree to remove Hadoop 2.7.3 and bump to 3.x for Iceberg 2.0.0.
>
> If we want to be based on Java 11+ (17 would be good imho), we have to
> check the dependencies. I'm not sure all of our dependencies support
> JDK17 today.
>
> Regards
> JB
>
> On Fri, Feb 16, 2024 at 10:09 AM Fokko Driesprong <fo...@apache.org>
> wrote:
> >
> > Hi everyone,
> >
> > I want to discuss adding the Hadoop upgrade to the list after moving to
> Iceberg 2.0. We still compile against Hadoop 2.7.3 to ensure we support as
> many users as possible. Hadoop 2.7.3 was released August 2016 and is not
> maintained anymore for a long time.
> >
> > My main reason for doing the upgrade is that on the Parquet MR project,
> I've been pushing back the Hadoop upgrade to ensure compatibility with
> Iceberg. However, at some point, we have to pull the trigger here. This
> will simplify things on the Parquet side and avoid having to check if the
> Java API Exists and such.
> >
> > Since Hadoop 3.3+ officially supports Java 11, I would suggest dropping
> everything below that. I wanted to check on the mailing list if there are
> any thoughts and or concerns.
> >
> > Kind regards,
> > Fokko
> >
> >
>
Re: Java Iceberg 2.0: Hadoop upgrade
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Fokko,
I agree to remove Hadoop 2.7.3 and bump to 3.x for Iceberg 2.0.0.
If we want to be based on Java 11+ (17 would be good imho), we have to
check the dependencies. I'm not sure all of our dependencies support
JDK17 today.
Regards
JB
On Fri, Feb 16, 2024 at 10:09 AM Fokko Driesprong <fo...@apache.org> wrote:
>
> Hi everyone,
>
> I want to discuss adding the Hadoop upgrade to the list after moving to Iceberg 2.0. We still compile against Hadoop 2.7.3 to ensure we support as many users as possible. Hadoop 2.7.3 was released August 2016 and is not maintained anymore for a long time.
>
> My main reason for doing the upgrade is that on the Parquet MR project, I've been pushing back the Hadoop upgrade to ensure compatibility with Iceberg. However, at some point, we have to pull the trigger here. This will simplify things on the Parquet side and avoid having to check if the Java API Exists and such.
>
> Since Hadoop 3.3+ officially supports Java 11, I would suggest dropping everything below that. I wanted to check on the mailing list if there are any thoughts and or concerns.
>
> Kind regards,
> Fokko
>
>
Re: Java Iceberg 2.0: Hadoop upgrade
Posted by Eduard Tudenhoefner <ed...@tabular.io>.
I would be +1 on dropping Hadoop 2.7.3 when moving to Iceberg 2.0.0. If
people want to use that Hadoop version, then there would still be the
Iceberg 1.x.y versions available for usage.
Another related topic would be the current Hive support, which essentially
makes it difficult for the project to upgrade to a newer JDK version, as
Hive requires JDK 8.
With Iceberg 2.0.0 an upgrade to JDK 17 (LTS) would be ideal, because even
right now a lot of dependencies and plugins that we use aren't maintained
anymore for JDK 8 and new (potentially critical) fixes are only available
in newer versions that are built for JDK 11/17.
Overall, I think it would make sense to discuss the scope of what should be
shipped with Iceberg 2.0.0 and where we'd drop support.
Eduard
On Fri, Feb 16, 2024 at 10:10 AM Fokko Driesprong <fo...@apache.org> wrote:
> Hi everyone,
>
> I want to discuss adding the Hadoop upgrade to the list after moving to
> Iceberg 2.0. We still compile against Hadoop 2.7.3 to ensure we support as
> many users as possible. Hadoop 2.7.3 was released August 2016
> <https://hadoop.apache.org/release/2.7.3.html> and is not
> maintained anymore <https://endoflife.date/apache-hadoop> for a long time.
>
> My main reason for doing the upgrade is that on the Parquet MR project,
> I've been pushing back the Hadoop upgrade to ensure compatibility with
> Iceberg. However, at some point, we have to pull the trigger here. This
> will simplify things on the Parquet side and avoid having to check if the
> Java API Exists and such.
>
> Since Hadoop 3.3+ officially supports Java 11
> <https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions>,
> I would suggest dropping everything below that. I wanted to check on the
> mailing list if there are any thoughts and or concerns.
>
> Kind regards,
> Fokko
>
>
>