You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Fokko Driesprong <fo...@apache.org> on 2024/11/04 08:42:28 UTC
[DISCUSS] Parquet 1.14.1 release
Hi everyone,
Cheng found a concerning bug
<https://github.com/apache/parquet-java/issues/3040> in Parquet and created
a fix immediately <https://github.com/apache/parquet-java/issues/3041>.
We should run a patch-release (1.14.1) to get this out to the public. Is
there anything else that we want to include?
Kind regards,
Fokko
Re: [DISCUSS] Parquet 1.14.1 release
Posted by Fokko Driesprong <fo...@apache.org>.
Hey everyone,
The Parquet sync provided some historical context, and the consensus was
that we should follow Symver. I've created a PR to add this to the Parquet
website <https://github.com/apache/parquet-site/pull/86>.
I'll continue to cherry-pick the commits to the maintenance branch for
1.14.4. Once 1.14.4 is released, we can discuss 1.15.0 to get all the new
stuff that has accumulated on the main branch out to the public. Let me
know if there are concerns.
Kind regards,
Fokko
Op di 5 nov 2024 om 16:11 schreef Fokko Driesprong <fo...@apache.org>:
> Hey everyone,
>
> In Iceberg 1.7.0, the Parquet dependency has been reverted from 1.14.3 to
> 1.13.1 to exclude the bug that Cheng pointed out. This way the release
> isn't blocked by the Parquet release.
>
> Fei, thanks for the PR. I noticed that this is not a bug fix bug but it
> adds a constructur. Typically in patch releases, only bug fixes are
> included. I couldn't find any guidance on the Parquet website, so not sure
> how rigid this is for Parquet. I would suggest discussing this in the
> Parquet sync tomorrow.
>
> Kind regards,
> Fokko
>
> Op di 5 nov 2024 om 04:47 schreef Wang, Yuming <yu...@ebay.com.invalid>:
>
>> Can we include this patch
>> https://github.com/apache/parquet-java/pull/3030.
>>
>> With this patch, after making some changes on the Spark side, the RPC
>> volume can be significantly reduced to improve the performance of the
>> entire cluster.
>>
>>
>>
>> *From: *Fokko Driesprong <fo...@apache.org>
>> *Date: *Monday, November 4, 2024 at 16:50
>> *To: *dev@parquet.apache.org <de...@parquet.apache.org>
>> *Subject: *Re: [DISCUSS] Parquet 1.14.1 release
>>
>> External Email
>>
>> Hey Gang,
>>
>> 1.14.4 indeed. I've created a milestone
>> <
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-java%2Fpull%2F3041&data=05%7C02%7Cyumwang%40ebay.com%7C9cb7ff15a1e5495aec7508dcfcadbf19%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638663070380205509%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=jx%2F%2FmJsLWUG12luShaYpz8OFNR3hd7JRPfiTPYh29U0%3D&reserved=0
>> <https://github.com/apache/parquet-java/pull/3041>> so we can track what
>> to
>> backport and added #3038 to it as well.
>>
>> Kind regards,
>> Fokko
>>
>> Op ma 4 nov 2024 om 09:46 schreef Gang Wu <us...@gmail.com>:
>>
>> > Hi Fokko,
>> >
>> > Thanks for bringing this up!
>> >
>> > I'd like to propose to include
>> >
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-java%2Fpull%2F3038&data=05%7C02%7Cyumwang%40ebay.com%7C9cb7ff15a1e5495aec7508dcfcadbf19%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638663070380231237%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=iOTBh5kbBwLDVagJrmoLEHzKqFaAtzS2%2FB3Z3VrZr1I%3D&reserved=0
>> <https://github.com/apache/parquet-java/pull/3038> as well.
>> >
>> > BTW, do you mean 1.14.4 instead of 1.14.1?
>> >
>> > Best,
>> > Gang
>> >
>> > On Mon, Nov 4, 2024 at 4:42 PM Fokko Driesprong <fo...@apache.org>
>> wrote:
>> >
>> > > Hi everyone,
>> > >
>> > > Cheng found a concerning bug
>> > > <
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-java%2Fissues%2F3040&data=05%7C02%7Cyumwang%40ebay.com%7C9cb7ff15a1e5495aec7508dcfcadbf19%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638663070380248982%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=x3vQG9VjHsQx9OnjHNPKQVsnpWt5yV5I566wjp9P6GU%3D&reserved=0
>> <https://github.com/apache/parquet-java/issues/3040>> in Parquet and
>> > > created
>> > > a fix immediately <
>> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-java%2Fissues%2F3041&data=05%7C02%7Cyumwang%40ebay.com%7C9cb7ff15a1e5495aec7508dcfcadbf19%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638663070380271192%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=wd40IOsh%2Ftwo3jXxuJ%2BUiYvX8YWd6wQisW7SxYo2Ags%3D&reserved=0
>> <https://github.com/apache/parquet-java/issues/3041>>.
>> > > We should run a patch-release (1.14.1) to get this out to the public.
>> Is
>> > > there anything else that we want to include?
>> > >
>> > > Kind regards,
>> > > Fokko
>> > >
>> >
>>
>
Re: [DISCUSS] Parquet 1.14.1 release
Posted by Fokko Driesprong <fo...@apache.org>.
Hey everyone,
In Iceberg 1.7.0, the Parquet dependency has been reverted from 1.14.3 to
1.13.1 to exclude the bug that Cheng pointed out. This way the release
isn't blocked by the Parquet release.
Fei, thanks for the PR. I noticed that this is not a bug fix bug but it
adds a constructur. Typically in patch releases, only bug fixes are
included. I couldn't find any guidance on the Parquet website, so not sure
how rigid this is for Parquet. I would suggest discussing this in the
Parquet sync tomorrow.
Kind regards,
Fokko
Op di 5 nov 2024 om 04:47 schreef Wang, Yuming <yu...@ebay.com.invalid>:
> Can we include this patch https://github.com/apache/parquet-java/pull/3030
> .
>
> With this patch, after making some changes on the Spark side, the RPC
> volume can be significantly reduced to improve the performance of the
> entire cluster.
>
>
>
> *From: *Fokko Driesprong <fo...@apache.org>
> *Date: *Monday, November 4, 2024 at 16:50
> *To: *dev@parquet.apache.org <de...@parquet.apache.org>
> *Subject: *Re: [DISCUSS] Parquet 1.14.1 release
>
> External Email
>
> Hey Gang,
>
> 1.14.4 indeed. I've created a milestone
> <
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-java%2Fpull%2F3041&data=05%7C02%7Cyumwang%40ebay.com%7C9cb7ff15a1e5495aec7508dcfcadbf19%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638663070380205509%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=jx%2F%2FmJsLWUG12luShaYpz8OFNR3hd7JRPfiTPYh29U0%3D&reserved=0
> <https://github.com/apache/parquet-java/pull/3041>> so we can track what
> to
> backport and added #3038 to it as well.
>
> Kind regards,
> Fokko
>
> Op ma 4 nov 2024 om 09:46 schreef Gang Wu <us...@gmail.com>:
>
> > Hi Fokko,
> >
> > Thanks for bringing this up!
> >
> > I'd like to propose to include
> >
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-java%2Fpull%2F3038&data=05%7C02%7Cyumwang%40ebay.com%7C9cb7ff15a1e5495aec7508dcfcadbf19%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638663070380231237%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=iOTBh5kbBwLDVagJrmoLEHzKqFaAtzS2%2FB3Z3VrZr1I%3D&reserved=0
> <https://github.com/apache/parquet-java/pull/3038> as well.
> >
> > BTW, do you mean 1.14.4 instead of 1.14.1?
> >
> > Best,
> > Gang
> >
> > On Mon, Nov 4, 2024 at 4:42 PM Fokko Driesprong <fo...@apache.org>
> wrote:
> >
> > > Hi everyone,
> > >
> > > Cheng found a concerning bug
> > > <
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-java%2Fissues%2F3040&data=05%7C02%7Cyumwang%40ebay.com%7C9cb7ff15a1e5495aec7508dcfcadbf19%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638663070380248982%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=x3vQG9VjHsQx9OnjHNPKQVsnpWt5yV5I566wjp9P6GU%3D&reserved=0
> <https://github.com/apache/parquet-java/issues/3040>> in Parquet and
> > > created
> > > a fix immediately <
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-java%2Fissues%2F3041&data=05%7C02%7Cyumwang%40ebay.com%7C9cb7ff15a1e5495aec7508dcfcadbf19%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638663070380271192%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=wd40IOsh%2Ftwo3jXxuJ%2BUiYvX8YWd6wQisW7SxYo2Ags%3D&reserved=0
> <https://github.com/apache/parquet-java/issues/3041>>.
> > > We should run a patch-release (1.14.1) to get this out to the public.
> Is
> > > there anything else that we want to include?
> > >
> > > Kind regards,
> > > Fokko
> > >
> >
>
Re: [DISCUSS] Parquet 1.14.1 release
Posted by "Wang, Yuming" <yu...@ebay.com.INVALID>.
Can we include this patch https://github.com/apache/parquet-java/pull/3030.
With this patch, after making some changes on the Spark side, the RPC volume can be significantly reduced to improve the performance of the entire cluster.
[cid:image001.png@01DB2F77.CCF2F5D0]
From: Fokko Driesprong <fo...@apache.org>
Date: Monday, November 4, 2024 at 16:50
To: dev@parquet.apache.org <de...@parquet.apache.org>
Subject: Re: [DISCUSS] Parquet 1.14.1 release
External Email
Hey Gang,
1.14.4 indeed. I've created a milestone
<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-java%2Fpull%2F3041&data=05%7C02%7Cyumwang%40ebay.com%7C9cb7ff15a1e5495aec7508dcfcadbf19%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638663070380205509%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=jx%2F%2FmJsLWUG12luShaYpz8OFNR3hd7JRPfiTPYh29U0%3D&reserved=0<https://github.com/apache/parquet-java/pull/3041>> so we can track what to
backport and added #3038 to it as well.
Kind regards,
Fokko
Op ma 4 nov 2024 om 09:46 schreef Gang Wu <us...@gmail.com>:
> Hi Fokko,
>
> Thanks for bringing this up!
>
> I'd like to propose to include
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-java%2Fpull%2F3038&data=05%7C02%7Cyumwang%40ebay.com%7C9cb7ff15a1e5495aec7508dcfcadbf19%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638663070380231237%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=iOTBh5kbBwLDVagJrmoLEHzKqFaAtzS2%2FB3Z3VrZr1I%3D&reserved=0<https://github.com/apache/parquet-java/pull/3038> as well.
>
> BTW, do you mean 1.14.4 instead of 1.14.1?
>
> Best,
> Gang
>
> On Mon, Nov 4, 2024 at 4:42 PM Fokko Driesprong <fo...@apache.org> wrote:
>
> > Hi everyone,
> >
> > Cheng found a concerning bug
> > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-java%2Fissues%2F3040&data=05%7C02%7Cyumwang%40ebay.com%7C9cb7ff15a1e5495aec7508dcfcadbf19%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638663070380248982%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=x3vQG9VjHsQx9OnjHNPKQVsnpWt5yV5I566wjp9P6GU%3D&reserved=0<https://github.com/apache/parquet-java/issues/3040>> in Parquet and
> > created
> > a fix immediately <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-java%2Fissues%2F3041&data=05%7C02%7Cyumwang%40ebay.com%7C9cb7ff15a1e5495aec7508dcfcadbf19%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C638663070380271192%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=wd40IOsh%2Ftwo3jXxuJ%2BUiYvX8YWd6wQisW7SxYo2Ags%3D&reserved=0<https://github.com/apache/parquet-java/issues/3041>>.
> > We should run a patch-release (1.14.1) to get this out to the public. Is
> > there anything else that we want to include?
> >
> > Kind regards,
> > Fokko
> >
>
Re: [DISCUSS] Parquet 1.14.1 release
Posted by Fokko Driesprong <fo...@apache.org>.
Hey Gang,
1.14.4 indeed. I've created a milestone
<https://github.com/apache/parquet-java/pull/3041> so we can track what to
backport and added #3038 to it as well.
Kind regards,
Fokko
Op ma 4 nov 2024 om 09:46 schreef Gang Wu <us...@gmail.com>:
> Hi Fokko,
>
> Thanks for bringing this up!
>
> I'd like to propose to include
> https://github.com/apache/parquet-java/pull/3038 as well.
>
> BTW, do you mean 1.14.4 instead of 1.14.1?
>
> Best,
> Gang
>
> On Mon, Nov 4, 2024 at 4:42 PM Fokko Driesprong <fo...@apache.org> wrote:
>
> > Hi everyone,
> >
> > Cheng found a concerning bug
> > <https://github.com/apache/parquet-java/issues/3040> in Parquet and
> > created
> > a fix immediately <https://github.com/apache/parquet-java/issues/3041>.
> > We should run a patch-release (1.14.1) to get this out to the public. Is
> > there anything else that we want to include?
> >
> > Kind regards,
> > Fokko
> >
>
Re: [DISCUSS] Parquet 1.14.1 release
Posted by Gang Wu <us...@gmail.com>.
Hi Fokko,
Thanks for bringing this up!
I'd like to propose to include
https://github.com/apache/parquet-java/pull/3038 as well.
BTW, do you mean 1.14.4 instead of 1.14.1?
Best,
Gang
On Mon, Nov 4, 2024 at 4:42 PM Fokko Driesprong <fo...@apache.org> wrote:
> Hi everyone,
>
> Cheng found a concerning bug
> <https://github.com/apache/parquet-java/issues/3040> in Parquet and
> created
> a fix immediately <https://github.com/apache/parquet-java/issues/3041>.
> We should run a patch-release (1.14.1) to get this out to the public. Is
> there anything else that we want to include?
>
> Kind regards,
> Fokko
>
Re: [DISCUSS] Parquet 1.14.1 release
Posted by Cheng Pan <pa...@gmail.com>.
Thanks Fokko for raising this discussion, yes, I agree that we need a patch release for 1.14 serial, but the version should be 1.14.4, right?
BTW, I confirm that the bug affects Spark 4.0.0-preview2, and I guess it also affects Iceberg 1.7.0 (under voting)
Thanks,
Cheng Pan
> On Nov 4, 2024, at 16:42, Fokko Driesprong <fo...@apache.org> wrote:
>
> Hi everyone,
>
> Cheng found a concerning bug
> <https://github.com/apache/parquet-java/issues/3040> in Parquet and created
> a fix immediately <https://github.com/apache/parquet-java/issues/3041>.
> We should run a patch-release (1.14.1) to get this out to the public. Is
> there anything else that we want to include?
>
> Kind regards,
> Fokko