You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Ismaël Mejía <ie...@gmail.com> on 2019/06/03 13:18:48 UTC
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Any chance the upgrade to Avro 1.9.0 may be cherry picked (otherwise
it will take longer to get this downstreamed into Spark).
I can work on the cherry pick if needed, just need to understand where
is the 1.11.x branch if we have one (or how is the procedure for this
in an ongoing release).
https://github.com/apache/parquet-mr/pull/638
On Fri, May 31, 2019 at 6:25 PM Ryan Blue <rb...@netflix.com.invalid> wrote:
>
> I think we can add that one.
>
> On Fri, May 31, 2019 at 9:18 AM Michael Heuer <he...@gmail.com> wrote:
>
> > Might
> >
> > https://github.com/apache/parquet-mr/pull/560
> >
> > be included in the next 1.11.0 release candidate?
> >
> > michael
> >
> >
> > On May 31, 2019, at 11:09 AM, Ryan Blue <rb...@netflix.com.INVALID> wrote:
> >
> > I'm hoping to find some time to get a release candidate out next week or
> > the week after.
> >
> > On Fri, May 31, 2019 at 4:32 AM Ismaël Mejía <ie...@apache.org> wrote:
> >
> > Hello,
> >
> > What is the status of this, any chance we can have 1.11.0 out soon?
> >
> > Thanks,
> > Ismaël
> >
> >
> > On 2019/03/19 14:00:30, Gabor Szadovszky <ga...@apache.org> wrote:
> >
> > Dear Parquet Users and Developers,
> >
> > I propose the following RC to be released as the official Apache
> > Parquet 1.11.0 release:
> >
> > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > * This corresponds to the tag: apache-parquet-1.11.0
> > *
> >
> >
> > https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
> >
> >
> > The release tarball, signature, and checksums are here:
> > *
> >
> > https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
> >
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/dev/parquet/KEYS
> >
> > Binary artifacts are staged in Nexus here:
> > *
> >
> >
> > https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
> >
> >
> > This release includes the following new features:
> > - PARQUET-1201 - Column indexes
> > - PARQUET-1253 - Support for new logical type representation
> > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
> >
> > The release also includes bug fixes, including:
> > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> > - PARQUET-1510: Fix notEq for optional columns with null values.
> > - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485
> >
> > change
> >
> > - PARQUET-1531: Page row count limit causes empty pages to be written
> >
> > from
> >
> > MessageColumnIO
> > - PARQUET-1544: Possible over-shading of modules
> >
> > The following change has been reverted so it is not part of any public
> > release:
> > - PARQUET-1381: Add merge blocks command to parquet-tools
> >
> > Please download, verify, and test. The vote will be open for at least 72
> > hours.
> >
> > Thanks,
> > Gabor
> >
> >
> >
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
> >
> >
> >
>
> --
> Ryan Blue
> Software Engineer
> Netflix
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Ismaël Mejía <ie...@gmail.com>.
Thanks Fokko,
It seems then that getting Avro 1.9.1 is a better idea, but in that
case the question is still the same, can this be backported on 1.11.x,
eventually for 1.11.1 ? I ask this because after looking at Parquet's
release pace I am worried that we will have to wait to long to get it
in.
On Mon, Jun 3, 2019 at 3:26 PM Driesprong, Fokko <fo...@driesprong.frl> wrote:
>
> Ismaël,
>
> From what I understand from Jacob Tolar, I believe we have a blocking issue
> in Apache Avro: https://issues.apache.org/jira/browse/AVRO-2400 We would
> need to release 1.9.1 first to get this sorted out.
>
> Regarding Parquet 1.11.0, I would expect a branch which has been cut from
> master. However, I don't see this branch on Github.
>
> Cheers, Fokko
>
> Op ma 3 jun. 2019 om 15:19 schreef Ismaël Mejía <ie...@gmail.com>:
>
> > Any chance the upgrade to Avro 1.9.0 may be cherry picked (otherwise
> > it will take longer to get this downstreamed into Spark).
> >
> > I can work on the cherry pick if needed, just need to understand where
> > is the 1.11.x branch if we have one (or how is the procedure for this
> > in an ongoing release).
> > https://github.com/apache/parquet-mr/pull/638
> >
> >
> >
> > On Fri, May 31, 2019 at 6:25 PM Ryan Blue <rb...@netflix.com.invalid>
> > wrote:
> > >
> > > I think we can add that one.
> > >
> > > On Fri, May 31, 2019 at 9:18 AM Michael Heuer <he...@gmail.com> wrote:
> > >
> > > > Might
> > > >
> > > > https://github.com/apache/parquet-mr/pull/560
> > > >
> > > > be included in the next 1.11.0 release candidate?
> > > >
> > > > michael
> > > >
> > > >
> > > > On May 31, 2019, at 11:09 AM, Ryan Blue <rb...@netflix.com.INVALID>
> > wrote:
> > > >
> > > > I'm hoping to find some time to get a release candidate out next week
> > or
> > > > the week after.
> > > >
> > > > On Fri, May 31, 2019 at 4:32 AM Ismaël Mejía <ie...@apache.org>
> > wrote:
> > > >
> > > > Hello,
> > > >
> > > > What is the status of this, any chance we can have 1.11.0 out soon?
> > > >
> > > > Thanks,
> > > > Ismaël
> > > >
> > > >
> > > > On 2019/03/19 14:00:30, Gabor Szadovszky <ga...@apache.org> wrote:
> > > >
> > > > Dear Parquet Users and Developers,
> > > >
> > > > I propose the following RC to be released as the official Apache
> > > > Parquet 1.11.0 release:
> > > >
> > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > *
> > > >
> > > >
> > > >
> > https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
> > > >
> > > >
> > > > The release tarball, signature, and checksums are here:
> > > > *
> > > >
> > > >
> > https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
> > > >
> > > >
> > > > You can find the KEYS file here:
> > > > * https://dist.apache.org/repos/dist/dev/parquet/KEYS
> > > >
> > > > Binary artifacts are staged in Nexus here:
> > > > *
> > > >
> > > >
> > > >
> > https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
> > > >
> > > >
> > > > This release includes the following new features:
> > > > - PARQUET-1201 - Column indexes
> > > > - PARQUET-1253 - Support for new logical type representation
> > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > > - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
> > > >
> > > > The release also includes bug fixes, including:
> > > > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> > > > - PARQUET-1510: Fix notEq for optional columns with null values.
> > > > - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485
> > > >
> > > > change
> > > >
> > > > - PARQUET-1531: Page row count limit causes empty pages to be written
> > > >
> > > > from
> > > >
> > > > MessageColumnIO
> > > > - PARQUET-1544: Possible over-shading of modules
> > > >
> > > > The following change has been reverted so it is not part of any public
> > > > release:
> > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > > >
> > > > Please download, verify, and test. The vote will be open for at least
> > 72
> > > > hours.
> > > >
> > > > Thanks,
> > > > Gabor
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Ryan Blue
> > > > Software Engineer
> > > > Netflix
> > > >
> > > >
> > > >
> > >
> > > --
> > > Ryan Blue
> > > Software Engineer
> > > Netflix
> >
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Ismaël,
From what I understand from Jacob Tolar, I believe we have a blocking issue
in Apache Avro: https://issues.apache.org/jira/browse/AVRO-2400 We would
need to release 1.9.1 first to get this sorted out.
Regarding Parquet 1.11.0, I would expect a branch which has been cut from
master. However, I don't see this branch on Github.
Cheers, Fokko
Op ma 3 jun. 2019 om 15:19 schreef Ismaël Mejía <ie...@gmail.com>:
> Any chance the upgrade to Avro 1.9.0 may be cherry picked (otherwise
> it will take longer to get this downstreamed into Spark).
>
> I can work on the cherry pick if needed, just need to understand where
> is the 1.11.x branch if we have one (or how is the procedure for this
> in an ongoing release).
> https://github.com/apache/parquet-mr/pull/638
>
>
>
> On Fri, May 31, 2019 at 6:25 PM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
> >
> > I think we can add that one.
> >
> > On Fri, May 31, 2019 at 9:18 AM Michael Heuer <he...@gmail.com> wrote:
> >
> > > Might
> > >
> > > https://github.com/apache/parquet-mr/pull/560
> > >
> > > be included in the next 1.11.0 release candidate?
> > >
> > > michael
> > >
> > >
> > > On May 31, 2019, at 11:09 AM, Ryan Blue <rb...@netflix.com.INVALID>
> wrote:
> > >
> > > I'm hoping to find some time to get a release candidate out next week
> or
> > > the week after.
> > >
> > > On Fri, May 31, 2019 at 4:32 AM Ismaël Mejía <ie...@apache.org>
> wrote:
> > >
> > > Hello,
> > >
> > > What is the status of this, any chance we can have 1.11.0 out soon?
> > >
> > > Thanks,
> > > Ismaël
> > >
> > >
> > > On 2019/03/19 14:00:30, Gabor Szadovszky <ga...@apache.org> wrote:
> > >
> > > Dear Parquet Users and Developers,
> > >
> > > I propose the following RC to be released as the official Apache
> > > Parquet 1.11.0 release:
> > >
> > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > * This corresponds to the tag: apache-parquet-1.11.0
> > > *
> > >
> > >
> > >
> https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
> > >
> > >
> > > The release tarball, signature, and checksums are here:
> > > *
> > >
> > >
> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
> > >
> > >
> > > You can find the KEYS file here:
> > > * https://dist.apache.org/repos/dist/dev/parquet/KEYS
> > >
> > > Binary artifacts are staged in Nexus here:
> > > *
> > >
> > >
> > >
> https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
> > >
> > >
> > > This release includes the following new features:
> > > - PARQUET-1201 - Column indexes
> > > - PARQUET-1253 - Support for new logical type representation
> > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
> > >
> > > The release also includes bug fixes, including:
> > > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> > > - PARQUET-1510: Fix notEq for optional columns with null values.
> > > - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485
> > >
> > > change
> > >
> > > - PARQUET-1531: Page row count limit causes empty pages to be written
> > >
> > > from
> > >
> > > MessageColumnIO
> > > - PARQUET-1544: Possible over-shading of modules
> > >
> > > The following change has been reverted so it is not part of any public
> > > release:
> > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > >
> > > Please download, verify, and test. The vote will be open for at least
> 72
> > > hours.
> > >
> > > Thanks,
> > > Gabor
> > >
> > >
> > >
> > >
> > > --
> > > Ryan Blue
> > > Software Engineer
> > > Netflix
> > >
> > >
> > >
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
>