You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Gabor Szadovszky <ga...@apache.org> on 2019/03/19 14:00:30 UTC
[VOTE] Release Apache Parquet 1.11.0 RC6
Dear Parquet Users and Developers,
I propose the following RC to be released as the official Apache
Parquet 1.11.0 release:
The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
* This corresponds to the tag: apache-parquet-1.11.0
*
https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/parquet/KEYS
Binary artifacts are staged in Nexus here:
*
https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
This release includes the following new features:
- PARQUET-1201 - Column indexes
- PARQUET-1253 - Support for new logical type representation
- PARQUET-1381 - Add merge blocks command to parquet-tools
- PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
The release also includes bug fixes, including:
- PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
- PARQUET-1510: Fix notEq for optional columns with null values.
- PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485 change
- PARQUET-1531: Page row count limit causes empty pages to be written from
MessageColumnIO
- PARQUET-1544: Possible over-shading of modules
The following change has been reverted so it is not part of any public
release:
- PARQUET-1381: Add merge blocks command to parquet-tools
Please download, verify, and test. The vote will be open for at least 72
hours.
Thanks,
Gabor
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Wes McKinney <we...@gmail.com>.
hi Zoltan,
I'm looking for ASF guidelines around this, whether it is MUST or SHOULD
https://www.apache.org/dev/release-signing#web-of-trust
Because SVN access is only password protected, having access to the
KEYS file is a weak standard of security. Could other PMC members
comment on this?
Thanks
On Mon, Apr 29, 2019 at 12:18 PM Zoltan Ivanfi <zi...@cloudera.com.invalid> wrote:
>
> Hi Wes,
>
> Gabor's key is in the KEYS file available at
> https://dist.apache.org/repos/dist/dev/parquet/KEYS Others may correct me
> if I'm mistaken, but as far as I know, this is all that is required. I
> mentioned this in the verification steps as well ("4. Verify the signature
> by running `gpg --verify apache-parquet-1.11.0.tar.gz.asc`. It should say
> "Good signature", the warning about the key not being trusted can be
> ignored"). My signing key is also unsigned, because instead of signing each
> other's keys we depend on the fact that only privileged users can put their
> key into the central KEYS file.
>
> Br,
>
> Zoltan
>
> On Mon, Apr 29, 2019 at 6:46 PM Wes McKinney <we...@gmail.com> wrote:
>
> > -1
> >
> > Gabor's PGP key is unsigned.
> >
> > $ gpg --verify apache-parquet-1.11.0.tar.gz.asc
> > gpg: assuming signed data in 'apache-parquet-1.11.0.tar.gz'
> > gpg: Signature made Tue 19 Mar 2019 08:55:48 AM CDT
> > gpg: using RSA key 6FB82970311551C7CEF131F5021057DBF048F543
> > gpg: Good signature from "Gabor Szadovszky <ga...@apache.org>" [unknown]
> > gpg: WARNING: This key is not certified with a trusted signature!
> > gpg: There is no indication that the signature belongs to the
> > owner.
> > Primary key fingerprint: 6FB8 2970 3115 51C7 CEF1 31F5 0210 57DB F048 F543
> >
> > On Tue, Apr 16, 2019 at 4:10 AM Gabor Szadovszky <ga...@apache.org> wrote:
> > >
> > > Based on our release process (
> > > http://parquet.apache.org/documentation/how-to-release/) and the related
> > > scripts we use the final tag for an RC. So, the existence of this tag
> > does
> > > not mean 1.11.0 is released.
> > > However, I agree this is misleading and not a good practice to remove
> > > already committed tags and re-add them to another place (when a new RC
> > > comes out). I think, we should update our release process to use RC tags
> > > and put the final tag only after it is officially released. But it is the
> > > story of the next release...
> > >
> > >
> > > On Sat, Apr 13, 2019 at 8:00 PM 俊杰陈 <cj...@gmail.com> wrote:
> > >
> > > > From the github release page, I see the 1.11.0 already released. Is it
> > > > still a rc version?
> > > >
> > https://github.com/apache/parquet-mr/releases/tag/apache-parquet-1.11.0
> > > >
> > > > On Fri, Apr 12, 2019 at 8:10 AM Ryan Blue <rb...@netflix.com.invalid>
> > > > wrote:
> > > >
> > > > > Personally, I haven't had enough time to devote to Parquet lately and
> > > > that
> > > > > means I haven't validated that this release's new features are okay
> > to
> > > > > release. I'm hoping sometime in the next few weeks I'll be able to
> > vote
> > > > on
> > > > > this.
> > > > >
> > > > > On Thu, Apr 11, 2019 at 1:23 PM Andy Grove <An...@rms.com>
> > wrote:
> > > > >
> > > > > > I'm curious if there is any update on this vote? The thread seems
> > > > eerily
> > > > > > quiet.
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > On 4/3/19, 10:38 AM, "Andy Grove" <An...@rms.com> wrote:
> > > > > >
> > > > > > CAUTION – UNVERIFIED EXTERNAL EMAIL
> > > > > >
> > > > > >
> > > > > > I have been able to run mvn verify and have also tested this RC
> > > > > > against our internal systems, with no issue.
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > I have raised the issue about Hadoop-lzo, but that is present
> > in
> > > > the
> > > > > > 1.10.1 release also.
> > > > > >
> > > > > > Andy.
> > > > > >
> > > > > >
> > > > > > On 3/20/19, 7:50 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID>
> > > > > wrote:
> > > > > >
> > > > > > CAUTION – UNVERIFIED EXTERNAL EMAIL
> > > > > >
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > signature matches
> > > > > > git hash matches the git tag
> > > > > > source tarball matches the git tag
> > > > > > unit tests and integration tests pass
> > > > > >
> > > > > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> > > > > gabor@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Dear Parquet Users and Developers,
> > > > > > >
> > > > > > > I propose the following RC to be released as the official
> > > > > Apache
> > > > > > > Parquet 1.11.0 release:
> > > > > > >
> > > > > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > > > > *
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=v6kHzIIpJQp%2Fq7fuR%2ByHVwGV7vZ7lUKupyqKZwmQeFI%3D&reserved=0
> > > > > > >
> > > > > > > The release tarball, signature, and checksums are here:
> > > > > > > *
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=RVlztCju4ZoZz5vnF8f5RxE7kPmZoKMj3Ipo4x0Aj4k%3D&reserved=0
> > > > > > >
> > > > > > > You can find the KEYS file here:
> > > > > > > *
> > > > > >
> > > > >
> > > >
> > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=8xPAIJ4EkJPXXxZ2hTH%2BuJOtCOrCspYXkjsl%2B44Jb20%3D&reserved=0
> > > > > > >
> > > > > > > Binary artifacts are staged in Nexus here:
> > > > > > > *
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342868310&sdata=%2FIW9qYFnwvuL7QgkrYxX%2BZWJ1fcaZz%2Bq1tRJWKfQERU%3D&reserved=0
> > > > > > >
> > > > > > > This release includes the following new features:
> > > > > > > - PARQUET-1201 - Column indexes
> > > > > > > - PARQUET-1253 - Support for new logical type
> > representation
> > > > > > > - PARQUET-1381 - Add merge blocks command to
> > parquet-tools
> > > > > > > - PARQUET-1388 - Nanosecond precision time and timestamp
> > -
> > > > > > parquet-mr
> > > > > > >
> > > > > > > The release also includes bug fixes, including:
> > > > > > > - PARQUET-1472: Dictionary filter fails on
> > > > > FIXED_LEN_BYTE_ARRAY.
> > > > > > > - PARQUET-1510: Fix notEq for optional columns with null
> > > > > values.
> > > > > > > - PARQUET-1533: TestSnappy() throws OOM exception with
> > > > > > Parquet-1485 change
> > > > > > > - PARQUET-1531: Page row count limit causes empty pages
> > to be
> > > > > > written from
> > > > > > > MessageColumnIO
> > > > > > > - PARQUET-1544: Possible over-shading of modules
> > > > > > >
> > > > > > > The following change has been reverted so it is not part
> > of
> > > > any
> > > > > > public
> > > > > > > release:
> > > > > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > > > > > >
> > > > > > > Please download, verify, and test. The vote will be open
> > for
> > > > at
> > > > > > least 72
> > > > > > > hours.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Gabor
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Ryan Blue
> > > > > Software Engineer
> > > > > Netflix
> > > > >
> > > >
> > > >
> > > > --
> > > > Thanks & Best Regards
> > > >
> >
Key signing (was: [VOTE] Release Apache Parquet 1.11.0 RC6)
Posted by Zoltan Ivanfi <zi...@cloudera.com.INVALID>.
Hi,
A video call sounds more secure to me than a photo which can be easily
manipulated. We could spend 5 minutes on it in the next Parquet sync
or alternatively is there someone already in the web of trust who
would volunteer to do a private video call with us before or after the
sync?
Thanks,
Zoltan
On Mon, Apr 29, 2019 at 7:52 PM Wes McKinney <we...@gmail.com> wrote:
>
> On Mon, Apr 29, 2019 at 12:48 PM Zoltan Ivanfi <zi...@cloudera.com.invalid> wrote:
> >
> > Hi,
> >
> > An excerpt from
> > https://www.apache.org/dev/release-signing#verifying-signature : "A
> > signature is valid, if gpg verifies the .asc as a good signature, and
> > doesn't complain about expired or revoked keys." Another excerpt from
> > https://www.apache.org/dev/release-signing#check-integrity that
> > reinforces that signing each other's keys is optional: "If you are
> > connected to the Apache web of trust then this also offers superior
> > security."
> >
> > That being said I support signing each other's keys. Of course, you
> > will still need one key somewhere along the signing chain that you
> > trust. I see that a few PMC members have signed keys, how should we
> > approach this task? The HOWTO suggests public conferences and key
> > signing parties, but I hope there is a way to do that remotely. Would
> > members who are already in the web of trust feel comfortable signing
> > our keys based the on the following?
> >
> > - Our keys have been committed to the central KEYS file using our
> > apache credentials.
> > - We could personally confirm this in the next Parquet sync.
> > - We could even read the key ID-s out loud if needed.
> >
>
> In person is best (if it is a person whose identity you are sure of),
> for people I know personally what I've done to sign their key remotely
> is have them write down the PGP fingerprint and show the paper to me
> in a photograph of themselves or in a video call. I don't know whether
> this is a good security practice but it seems better than doing things
> over e-mail =)
>
> - Wes
>
> > Br,
> >
> > Zoltan
> >
> >
> > On Mon, Apr 29, 2019 at 7:11 PM Zoltan Ivanfi <zi...@cloudera.com> wrote:
> > >
> > > Hi Wes,
> > >
> > > Gabor's key is in the KEYS file available at https://dist.apache.org/repos/dist/dev/parquet/KEYS Others may correct me if I'm mistaken, but as far as I know, this is all that is required. I mentioned this in the verification steps as well ("4. Verify the signature by running `gpg --verify apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature", the warning about the key not being trusted can be ignored"). My signing key is also unsigned, because instead of signing each other's keys we depend on the fact that only privileged users can put their key into the central KEYS file.
> > >
> > > Br,
> > >
> > > Zoltan
> > >
> > > On Mon, Apr 29, 2019 at 6:46 PM Wes McKinney <we...@gmail.com> wrote:
> > >>
> > >> -1
> > >>
> > >> Gabor's PGP key is unsigned.
> > >>
> > >> $ gpg --verify apache-parquet-1.11.0.tar.gz.asc
> > >> gpg: assuming signed data in 'apache-parquet-1.11.0.tar.gz'
> > >> gpg: Signature made Tue 19 Mar 2019 08:55:48 AM CDT
> > >> gpg: using RSA key 6FB82970311551C7CEF131F5021057DBF048F543
> > >> gpg: Good signature from "Gabor Szadovszky <ga...@apache.org>" [unknown]
> > >> gpg: WARNING: This key is not certified with a trusted signature!
> > >> gpg: There is no indication that the signature belongs to the owner.
> > >> Primary key fingerprint: 6FB8 2970 3115 51C7 CEF1 31F5 0210 57DB F048 F543
> > >>
> > >> On Tue, Apr 16, 2019 at 4:10 AM Gabor Szadovszky <ga...@apache.org> wrote:
> > >> >
> > >> > Based on our release process (
> > >> > http://parquet.apache.org/documentation/how-to-release/) and the related
> > >> > scripts we use the final tag for an RC. So, the existence of this tag does
> > >> > not mean 1.11.0 is released.
> > >> > However, I agree this is misleading and not a good practice to remove
> > >> > already committed tags and re-add them to another place (when a new RC
> > >> > comes out). I think, we should update our release process to use RC tags
> > >> > and put the final tag only after it is officially released. But it is the
> > >> > story of the next release...
> > >> >
> > >> >
> > >> > On Sat, Apr 13, 2019 at 8:00 PM 俊杰陈 <cj...@gmail.com> wrote:
> > >> >
> > >> > > From the github release page, I see the 1.11.0 already released. Is it
> > >> > > still a rc version?
> > >> > > https://github.com/apache/parquet-mr/releases/tag/apache-parquet-1.11.0
> > >> > >
> > >> > > On Fri, Apr 12, 2019 at 8:10 AM Ryan Blue <rb...@netflix.com.invalid>
> > >> > > wrote:
> > >> > >
> > >> > > > Personally, I haven't had enough time to devote to Parquet lately and
> > >> > > that
> > >> > > > means I haven't validated that this release's new features are okay to
> > >> > > > release. I'm hoping sometime in the next few weeks I'll be able to vote
> > >> > > on
> > >> > > > this.
> > >> > > >
> > >> > > > On Thu, Apr 11, 2019 at 1:23 PM Andy Grove <An...@rms.com> wrote:
> > >> > > >
> > >> > > > > I'm curious if there is any update on this vote? The thread seems
> > >> > > eerily
> > >> > > > > quiet.
> > >> > > > >
> > >> > > > > Thanks.
> > >> > > > >
> > >> > > > > On 4/3/19, 10:38 AM, "Andy Grove" <An...@rms.com> wrote:
> > >> > > > >
> > >> > > > > CAUTION – UNVERIFIED EXTERNAL EMAIL
> > >> > > > >
> > >> > > > >
> > >> > > > > I have been able to run mvn verify and have also tested this RC
> > >> > > > > against our internal systems, with no issue.
> > >> > > > >
> > >> > > > > +1 (non-binding)
> > >> > > > >
> > >> > > > > I have raised the issue about Hadoop-lzo, but that is present in
> > >> > > the
> > >> > > > > 1.10.1 release also.
> > >> > > > >
> > >> > > > > Andy.
> > >> > > > >
> > >> > > > >
> > >> > > > > On 3/20/19, 7:50 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID>
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > CAUTION – UNVERIFIED EXTERNAL EMAIL
> > >> > > > >
> > >> > > > >
> > >> > > > > +1 (binding)
> > >> > > > >
> > >> > > > > signature matches
> > >> > > > > git hash matches the git tag
> > >> > > > > source tarball matches the git tag
> > >> > > > > unit tests and integration tests pass
> > >> > > > >
> > >> > > > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> > >> > > > gabor@apache.org>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Dear Parquet Users and Developers,
> > >> > > > > >
> > >> > > > > > I propose the following RC to be released as the official
> > >> > > > Apache
> > >> > > > > > Parquet 1.11.0 release:
> > >> > > > > >
> > >> > > > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > >> > > > > > * This corresponds to the tag: apache-parquet-1.11.0
> > >> > > > > > *
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=v6kHzIIpJQp%2Fq7fuR%2ByHVwGV7vZ7lUKupyqKZwmQeFI%3D&reserved=0
> > >> > > > > >
> > >> > > > > > The release tarball, signature, and checksums are here:
> > >> > > > > > *
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=RVlztCju4ZoZz5vnF8f5RxE7kPmZoKMj3Ipo4x0Aj4k%3D&reserved=0
> > >> > > > > >
> > >> > > > > > You can find the KEYS file here:
> > >> > > > > > *
> > >> > > > >
> > >> > > >
> > >> > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=8xPAIJ4EkJPXXxZ2hTH%2BuJOtCOrCspYXkjsl%2B44Jb20%3D&reserved=0
> > >> > > > > >
> > >> > > > > > Binary artifacts are staged in Nexus here:
> > >> > > > > > *
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342868310&sdata=%2FIW9qYFnwvuL7QgkrYxX%2BZWJ1fcaZz%2Bq1tRJWKfQERU%3D&reserved=0
> > >> > > > > >
> > >> > > > > > This release includes the following new features:
> > >> > > > > > - PARQUET-1201 - Column indexes
> > >> > > > > > - PARQUET-1253 - Support for new logical type representation
> > >> > > > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > >> > > > > > - PARQUET-1388 - Nanosecond precision time and timestamp -
> > >> > > > > parquet-mr
> > >> > > > > >
> > >> > > > > > The release also includes bug fixes, including:
> > >> > > > > > - PARQUET-1472: Dictionary filter fails on
> > >> > > > FIXED_LEN_BYTE_ARRAY.
> > >> > > > > > - PARQUET-1510: Fix notEq for optional columns with null
> > >> > > > values.
> > >> > > > > > - PARQUET-1533: TestSnappy() throws OOM exception with
> > >> > > > > Parquet-1485 change
> > >> > > > > > - PARQUET-1531: Page row count limit causes empty pages to be
> > >> > > > > written from
> > >> > > > > > MessageColumnIO
> > >> > > > > > - PARQUET-1544: Possible over-shading of modules
> > >> > > > > >
> > >> > > > > > The following change has been reverted so it is not part of
> > >> > > any
> > >> > > > > public
> > >> > > > > > release:
> > >> > > > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > >> > > > > >
> > >> > > > > > Please download, verify, and test. The vote will be open for
> > >> > > at
> > >> > > > > least 72
> > >> > > > > > hours.
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Gabor
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > > > --
> > >> > > > Ryan Blue
> > >> > > > Software Engineer
> > >> > > > Netflix
> > >> > > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Thanks & Best Regards
> > >> > >
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Wes McKinney <we...@gmail.com>.
On Mon, Apr 29, 2019 at 12:48 PM Zoltan Ivanfi <zi...@cloudera.com.invalid> wrote:
>
> Hi,
>
> An excerpt from
> https://www.apache.org/dev/release-signing#verifying-signature : "A
> signature is valid, if gpg verifies the .asc as a good signature, and
> doesn't complain about expired or revoked keys." Another excerpt from
> https://www.apache.org/dev/release-signing#check-integrity that
> reinforces that signing each other's keys is optional: "If you are
> connected to the Apache web of trust then this also offers superior
> security."
>
> That being said I support signing each other's keys. Of course, you
> will still need one key somewhere along the signing chain that you
> trust. I see that a few PMC members have signed keys, how should we
> approach this task? The HOWTO suggests public conferences and key
> signing parties, but I hope there is a way to do that remotely. Would
> members who are already in the web of trust feel comfortable signing
> our keys based the on the following?
>
> - Our keys have been committed to the central KEYS file using our
> apache credentials.
> - We could personally confirm this in the next Parquet sync.
> - We could even read the key ID-s out loud if needed.
>
In person is best (if it is a person whose identity you are sure of),
for people I know personally what I've done to sign their key remotely
is have them write down the PGP fingerprint and show the paper to me
in a photograph of themselves or in a video call. I don't know whether
this is a good security practice but it seems better than doing things
over e-mail =)
- Wes
> Br,
>
> Zoltan
>
>
> On Mon, Apr 29, 2019 at 7:11 PM Zoltan Ivanfi <zi...@cloudera.com> wrote:
> >
> > Hi Wes,
> >
> > Gabor's key is in the KEYS file available at https://dist.apache.org/repos/dist/dev/parquet/KEYS Others may correct me if I'm mistaken, but as far as I know, this is all that is required. I mentioned this in the verification steps as well ("4. Verify the signature by running `gpg --verify apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature", the warning about the key not being trusted can be ignored"). My signing key is also unsigned, because instead of signing each other's keys we depend on the fact that only privileged users can put their key into the central KEYS file.
> >
> > Br,
> >
> > Zoltan
> >
> > On Mon, Apr 29, 2019 at 6:46 PM Wes McKinney <we...@gmail.com> wrote:
> >>
> >> -1
> >>
> >> Gabor's PGP key is unsigned.
> >>
> >> $ gpg --verify apache-parquet-1.11.0.tar.gz.asc
> >> gpg: assuming signed data in 'apache-parquet-1.11.0.tar.gz'
> >> gpg: Signature made Tue 19 Mar 2019 08:55:48 AM CDT
> >> gpg: using RSA key 6FB82970311551C7CEF131F5021057DBF048F543
> >> gpg: Good signature from "Gabor Szadovszky <ga...@apache.org>" [unknown]
> >> gpg: WARNING: This key is not certified with a trusted signature!
> >> gpg: There is no indication that the signature belongs to the owner.
> >> Primary key fingerprint: 6FB8 2970 3115 51C7 CEF1 31F5 0210 57DB F048 F543
> >>
> >> On Tue, Apr 16, 2019 at 4:10 AM Gabor Szadovszky <ga...@apache.org> wrote:
> >> >
> >> > Based on our release process (
> >> > http://parquet.apache.org/documentation/how-to-release/) and the related
> >> > scripts we use the final tag for an RC. So, the existence of this tag does
> >> > not mean 1.11.0 is released.
> >> > However, I agree this is misleading and not a good practice to remove
> >> > already committed tags and re-add them to another place (when a new RC
> >> > comes out). I think, we should update our release process to use RC tags
> >> > and put the final tag only after it is officially released. But it is the
> >> > story of the next release...
> >> >
> >> >
> >> > On Sat, Apr 13, 2019 at 8:00 PM 俊杰陈 <cj...@gmail.com> wrote:
> >> >
> >> > > From the github release page, I see the 1.11.0 already released. Is it
> >> > > still a rc version?
> >> > > https://github.com/apache/parquet-mr/releases/tag/apache-parquet-1.11.0
> >> > >
> >> > > On Fri, Apr 12, 2019 at 8:10 AM Ryan Blue <rb...@netflix.com.invalid>
> >> > > wrote:
> >> > >
> >> > > > Personally, I haven't had enough time to devote to Parquet lately and
> >> > > that
> >> > > > means I haven't validated that this release's new features are okay to
> >> > > > release. I'm hoping sometime in the next few weeks I'll be able to vote
> >> > > on
> >> > > > this.
> >> > > >
> >> > > > On Thu, Apr 11, 2019 at 1:23 PM Andy Grove <An...@rms.com> wrote:
> >> > > >
> >> > > > > I'm curious if there is any update on this vote? The thread seems
> >> > > eerily
> >> > > > > quiet.
> >> > > > >
> >> > > > > Thanks.
> >> > > > >
> >> > > > > On 4/3/19, 10:38 AM, "Andy Grove" <An...@rms.com> wrote:
> >> > > > >
> >> > > > > CAUTION – UNVERIFIED EXTERNAL EMAIL
> >> > > > >
> >> > > > >
> >> > > > > I have been able to run mvn verify and have also tested this RC
> >> > > > > against our internal systems, with no issue.
> >> > > > >
> >> > > > > +1 (non-binding)
> >> > > > >
> >> > > > > I have raised the issue about Hadoop-lzo, but that is present in
> >> > > the
> >> > > > > 1.10.1 release also.
> >> > > > >
> >> > > > > Andy.
> >> > > > >
> >> > > > >
> >> > > > > On 3/20/19, 7:50 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID>
> >> > > > wrote:
> >> > > > >
> >> > > > > CAUTION – UNVERIFIED EXTERNAL EMAIL
> >> > > > >
> >> > > > >
> >> > > > > +1 (binding)
> >> > > > >
> >> > > > > signature matches
> >> > > > > git hash matches the git tag
> >> > > > > source tarball matches the git tag
> >> > > > > unit tests and integration tests pass
> >> > > > >
> >> > > > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> >> > > > gabor@apache.org>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Dear Parquet Users and Developers,
> >> > > > > >
> >> > > > > > I propose the following RC to be released as the official
> >> > > > Apache
> >> > > > > > Parquet 1.11.0 release:
> >> > > > > >
> >> > > > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> >> > > > > > * This corresponds to the tag: apache-parquet-1.11.0
> >> > > > > > *
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=v6kHzIIpJQp%2Fq7fuR%2ByHVwGV7vZ7lUKupyqKZwmQeFI%3D&reserved=0
> >> > > > > >
> >> > > > > > The release tarball, signature, and checksums are here:
> >> > > > > > *
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=RVlztCju4ZoZz5vnF8f5RxE7kPmZoKMj3Ipo4x0Aj4k%3D&reserved=0
> >> > > > > >
> >> > > > > > You can find the KEYS file here:
> >> > > > > > *
> >> > > > >
> >> > > >
> >> > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=8xPAIJ4EkJPXXxZ2hTH%2BuJOtCOrCspYXkjsl%2B44Jb20%3D&reserved=0
> >> > > > > >
> >> > > > > > Binary artifacts are staged in Nexus here:
> >> > > > > > *
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342868310&sdata=%2FIW9qYFnwvuL7QgkrYxX%2BZWJ1fcaZz%2Bq1tRJWKfQERU%3D&reserved=0
> >> > > > > >
> >> > > > > > This release includes the following new features:
> >> > > > > > - PARQUET-1201 - Column indexes
> >> > > > > > - PARQUET-1253 - Support for new logical type representation
> >> > > > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> >> > > > > > - PARQUET-1388 - Nanosecond precision time and timestamp -
> >> > > > > parquet-mr
> >> > > > > >
> >> > > > > > The release also includes bug fixes, including:
> >> > > > > > - PARQUET-1472: Dictionary filter fails on
> >> > > > FIXED_LEN_BYTE_ARRAY.
> >> > > > > > - PARQUET-1510: Fix notEq for optional columns with null
> >> > > > values.
> >> > > > > > - PARQUET-1533: TestSnappy() throws OOM exception with
> >> > > > > Parquet-1485 change
> >> > > > > > - PARQUET-1531: Page row count limit causes empty pages to be
> >> > > > > written from
> >> > > > > > MessageColumnIO
> >> > > > > > - PARQUET-1544: Possible over-shading of modules
> >> > > > > >
> >> > > > > > The following change has been reverted so it is not part of
> >> > > any
> >> > > > > public
> >> > > > > > release:
> >> > > > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> >> > > > > >
> >> > > > > > Please download, verify, and test. The vote will be open for
> >> > > at
> >> > > > > least 72
> >> > > > > > hours.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Gabor
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > > --
> >> > > > Ryan Blue
> >> > > > Software Engineer
> >> > > > Netflix
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > Thanks & Best Regards
> >> > >
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Zoltan Ivanfi <zi...@cloudera.com.INVALID>.
Hi,
An excerpt from
https://www.apache.org/dev/release-signing#verifying-signature : "A
signature is valid, if gpg verifies the .asc as a good signature, and
doesn't complain about expired or revoked keys." Another excerpt from
https://www.apache.org/dev/release-signing#check-integrity that
reinforces that signing each other's keys is optional: "If you are
connected to the Apache web of trust then this also offers superior
security."
That being said I support signing each other's keys. Of course, you
will still need one key somewhere along the signing chain that you
trust. I see that a few PMC members have signed keys, how should we
approach this task? The HOWTO suggests public conferences and key
signing parties, but I hope there is a way to do that remotely. Would
members who are already in the web of trust feel comfortable signing
our keys based the on the following?
- Our keys have been committed to the central KEYS file using our
apache credentials.
- We could personally confirm this in the next Parquet sync.
- We could even read the key ID-s out loud if needed.
Br,
Zoltan
On Mon, Apr 29, 2019 at 7:11 PM Zoltan Ivanfi <zi...@cloudera.com> wrote:
>
> Hi Wes,
>
> Gabor's key is in the KEYS file available at https://dist.apache.org/repos/dist/dev/parquet/KEYS Others may correct me if I'm mistaken, but as far as I know, this is all that is required. I mentioned this in the verification steps as well ("4. Verify the signature by running `gpg --verify apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature", the warning about the key not being trusted can be ignored"). My signing key is also unsigned, because instead of signing each other's keys we depend on the fact that only privileged users can put their key into the central KEYS file.
>
> Br,
>
> Zoltan
>
> On Mon, Apr 29, 2019 at 6:46 PM Wes McKinney <we...@gmail.com> wrote:
>>
>> -1
>>
>> Gabor's PGP key is unsigned.
>>
>> $ gpg --verify apache-parquet-1.11.0.tar.gz.asc
>> gpg: assuming signed data in 'apache-parquet-1.11.0.tar.gz'
>> gpg: Signature made Tue 19 Mar 2019 08:55:48 AM CDT
>> gpg: using RSA key 6FB82970311551C7CEF131F5021057DBF048F543
>> gpg: Good signature from "Gabor Szadovszky <ga...@apache.org>" [unknown]
>> gpg: WARNING: This key is not certified with a trusted signature!
>> gpg: There is no indication that the signature belongs to the owner.
>> Primary key fingerprint: 6FB8 2970 3115 51C7 CEF1 31F5 0210 57DB F048 F543
>>
>> On Tue, Apr 16, 2019 at 4:10 AM Gabor Szadovszky <ga...@apache.org> wrote:
>> >
>> > Based on our release process (
>> > http://parquet.apache.org/documentation/how-to-release/) and the related
>> > scripts we use the final tag for an RC. So, the existence of this tag does
>> > not mean 1.11.0 is released.
>> > However, I agree this is misleading and not a good practice to remove
>> > already committed tags and re-add them to another place (when a new RC
>> > comes out). I think, we should update our release process to use RC tags
>> > and put the final tag only after it is officially released. But it is the
>> > story of the next release...
>> >
>> >
>> > On Sat, Apr 13, 2019 at 8:00 PM 俊杰陈 <cj...@gmail.com> wrote:
>> >
>> > > From the github release page, I see the 1.11.0 already released. Is it
>> > > still a rc version?
>> > > https://github.com/apache/parquet-mr/releases/tag/apache-parquet-1.11.0
>> > >
>> > > On Fri, Apr 12, 2019 at 8:10 AM Ryan Blue <rb...@netflix.com.invalid>
>> > > wrote:
>> > >
>> > > > Personally, I haven't had enough time to devote to Parquet lately and
>> > > that
>> > > > means I haven't validated that this release's new features are okay to
>> > > > release. I'm hoping sometime in the next few weeks I'll be able to vote
>> > > on
>> > > > this.
>> > > >
>> > > > On Thu, Apr 11, 2019 at 1:23 PM Andy Grove <An...@rms.com> wrote:
>> > > >
>> > > > > I'm curious if there is any update on this vote? The thread seems
>> > > eerily
>> > > > > quiet.
>> > > > >
>> > > > > Thanks.
>> > > > >
>> > > > > On 4/3/19, 10:38 AM, "Andy Grove" <An...@rms.com> wrote:
>> > > > >
>> > > > > CAUTION – UNVERIFIED EXTERNAL EMAIL
>> > > > >
>> > > > >
>> > > > > I have been able to run mvn verify and have also tested this RC
>> > > > > against our internal systems, with no issue.
>> > > > >
>> > > > > +1 (non-binding)
>> > > > >
>> > > > > I have raised the issue about Hadoop-lzo, but that is present in
>> > > the
>> > > > > 1.10.1 release also.
>> > > > >
>> > > > > Andy.
>> > > > >
>> > > > >
>> > > > > On 3/20/19, 7:50 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID>
>> > > > wrote:
>> > > > >
>> > > > > CAUTION – UNVERIFIED EXTERNAL EMAIL
>> > > > >
>> > > > >
>> > > > > +1 (binding)
>> > > > >
>> > > > > signature matches
>> > > > > git hash matches the git tag
>> > > > > source tarball matches the git tag
>> > > > > unit tests and integration tests pass
>> > > > >
>> > > > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
>> > > > gabor@apache.org>
>> > > > > wrote:
>> > > > >
>> > > > > > Dear Parquet Users and Developers,
>> > > > > >
>> > > > > > I propose the following RC to be released as the official
>> > > > Apache
>> > > > > > Parquet 1.11.0 release:
>> > > > > >
>> > > > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
>> > > > > > * This corresponds to the tag: apache-parquet-1.11.0
>> > > > > > *
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=v6kHzIIpJQp%2Fq7fuR%2ByHVwGV7vZ7lUKupyqKZwmQeFI%3D&reserved=0
>> > > > > >
>> > > > > > The release tarball, signature, and checksums are here:
>> > > > > > *
>> > > > > >
>> > > > >
>> > > >
>> > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=RVlztCju4ZoZz5vnF8f5RxE7kPmZoKMj3Ipo4x0Aj4k%3D&reserved=0
>> > > > > >
>> > > > > > You can find the KEYS file here:
>> > > > > > *
>> > > > >
>> > > >
>> > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=8xPAIJ4EkJPXXxZ2hTH%2BuJOtCOrCspYXkjsl%2B44Jb20%3D&reserved=0
>> > > > > >
>> > > > > > Binary artifacts are staged in Nexus here:
>> > > > > > *
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342868310&sdata=%2FIW9qYFnwvuL7QgkrYxX%2BZWJ1fcaZz%2Bq1tRJWKfQERU%3D&reserved=0
>> > > > > >
>> > > > > > This release includes the following new features:
>> > > > > > - PARQUET-1201 - Column indexes
>> > > > > > - PARQUET-1253 - Support for new logical type representation
>> > > > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
>> > > > > > - PARQUET-1388 - Nanosecond precision time and timestamp -
>> > > > > parquet-mr
>> > > > > >
>> > > > > > The release also includes bug fixes, including:
>> > > > > > - PARQUET-1472: Dictionary filter fails on
>> > > > FIXED_LEN_BYTE_ARRAY.
>> > > > > > - PARQUET-1510: Fix notEq for optional columns with null
>> > > > values.
>> > > > > > - PARQUET-1533: TestSnappy() throws OOM exception with
>> > > > > Parquet-1485 change
>> > > > > > - PARQUET-1531: Page row count limit causes empty pages to be
>> > > > > written from
>> > > > > > MessageColumnIO
>> > > > > > - PARQUET-1544: Possible over-shading of modules
>> > > > > >
>> > > > > > The following change has been reverted so it is not part of
>> > > any
>> > > > > public
>> > > > > > release:
>> > > > > > - PARQUET-1381: Add merge blocks command to parquet-tools
>> > > > > >
>> > > > > > Please download, verify, and test. The vote will be open for
>> > > at
>> > > > > least 72
>> > > > > > hours.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Gabor
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > > --
>> > > > Ryan Blue
>> > > > Software Engineer
>> > > > Netflix
>> > > >
>> > >
>> > >
>> > > --
>> > > Thanks & Best Regards
>> > >
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Zoltan Ivanfi <zi...@cloudera.com.INVALID>.
Hi Wes,
Gabor's key is in the KEYS file available at
https://dist.apache.org/repos/dist/dev/parquet/KEYS Others may correct me
if I'm mistaken, but as far as I know, this is all that is required. I
mentioned this in the verification steps as well ("4. Verify the signature
by running `gpg --verify apache-parquet-1.11.0.tar.gz.asc`. It should say
"Good signature", the warning about the key not being trusted can be
ignored"). My signing key is also unsigned, because instead of signing each
other's keys we depend on the fact that only privileged users can put their
key into the central KEYS file.
Br,
Zoltan
On Mon, Apr 29, 2019 at 6:46 PM Wes McKinney <we...@gmail.com> wrote:
> -1
>
> Gabor's PGP key is unsigned.
>
> $ gpg --verify apache-parquet-1.11.0.tar.gz.asc
> gpg: assuming signed data in 'apache-parquet-1.11.0.tar.gz'
> gpg: Signature made Tue 19 Mar 2019 08:55:48 AM CDT
> gpg: using RSA key 6FB82970311551C7CEF131F5021057DBF048F543
> gpg: Good signature from "Gabor Szadovszky <ga...@apache.org>" [unknown]
> gpg: WARNING: This key is not certified with a trusted signature!
> gpg: There is no indication that the signature belongs to the
> owner.
> Primary key fingerprint: 6FB8 2970 3115 51C7 CEF1 31F5 0210 57DB F048 F543
>
> On Tue, Apr 16, 2019 at 4:10 AM Gabor Szadovszky <ga...@apache.org> wrote:
> >
> > Based on our release process (
> > http://parquet.apache.org/documentation/how-to-release/) and the related
> > scripts we use the final tag for an RC. So, the existence of this tag
> does
> > not mean 1.11.0 is released.
> > However, I agree this is misleading and not a good practice to remove
> > already committed tags and re-add them to another place (when a new RC
> > comes out). I think, we should update our release process to use RC tags
> > and put the final tag only after it is officially released. But it is the
> > story of the next release...
> >
> >
> > On Sat, Apr 13, 2019 at 8:00 PM 俊杰陈 <cj...@gmail.com> wrote:
> >
> > > From the github release page, I see the 1.11.0 already released. Is it
> > > still a rc version?
> > >
> https://github.com/apache/parquet-mr/releases/tag/apache-parquet-1.11.0
> > >
> > > On Fri, Apr 12, 2019 at 8:10 AM Ryan Blue <rb...@netflix.com.invalid>
> > > wrote:
> > >
> > > > Personally, I haven't had enough time to devote to Parquet lately and
> > > that
> > > > means I haven't validated that this release's new features are okay
> to
> > > > release. I'm hoping sometime in the next few weeks I'll be able to
> vote
> > > on
> > > > this.
> > > >
> > > > On Thu, Apr 11, 2019 at 1:23 PM Andy Grove <An...@rms.com>
> wrote:
> > > >
> > > > > I'm curious if there is any update on this vote? The thread seems
> > > eerily
> > > > > quiet.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > On 4/3/19, 10:38 AM, "Andy Grove" <An...@rms.com> wrote:
> > > > >
> > > > > CAUTION – UNVERIFIED EXTERNAL EMAIL
> > > > >
> > > > >
> > > > > I have been able to run mvn verify and have also tested this RC
> > > > > against our internal systems, with no issue.
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > I have raised the issue about Hadoop-lzo, but that is present
> in
> > > the
> > > > > 1.10.1 release also.
> > > > >
> > > > > Andy.
> > > > >
> > > > >
> > > > > On 3/20/19, 7:50 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID>
> > > > wrote:
> > > > >
> > > > > CAUTION – UNVERIFIED EXTERNAL EMAIL
> > > > >
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > signature matches
> > > > > git hash matches the git tag
> > > > > source tarball matches the git tag
> > > > > unit tests and integration tests pass
> > > > >
> > > > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> > > > gabor@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Dear Parquet Users and Developers,
> > > > > >
> > > > > > I propose the following RC to be released as the official
> > > > Apache
> > > > > > Parquet 1.11.0 release:
> > > > > >
> > > > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > > > *
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=v6kHzIIpJQp%2Fq7fuR%2ByHVwGV7vZ7lUKupyqKZwmQeFI%3D&reserved=0
> > > > > >
> > > > > > The release tarball, signature, and checksums are here:
> > > > > > *
> > > > > >
> > > > >
> > > >
> > >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=RVlztCju4ZoZz5vnF8f5RxE7kPmZoKMj3Ipo4x0Aj4k%3D&reserved=0
> > > > > >
> > > > > > You can find the KEYS file here:
> > > > > > *
> > > > >
> > > >
> > >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=8xPAIJ4EkJPXXxZ2hTH%2BuJOtCOrCspYXkjsl%2B44Jb20%3D&reserved=0
> > > > > >
> > > > > > Binary artifacts are staged in Nexus here:
> > > > > > *
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342868310&sdata=%2FIW9qYFnwvuL7QgkrYxX%2BZWJ1fcaZz%2Bq1tRJWKfQERU%3D&reserved=0
> > > > > >
> > > > > > This release includes the following new features:
> > > > > > - PARQUET-1201 - Column indexes
> > > > > > - PARQUET-1253 - Support for new logical type
> representation
> > > > > > - PARQUET-1381 - Add merge blocks command to
> parquet-tools
> > > > > > - PARQUET-1388 - Nanosecond precision time and timestamp
> -
> > > > > parquet-mr
> > > > > >
> > > > > > The release also includes bug fixes, including:
> > > > > > - PARQUET-1472: Dictionary filter fails on
> > > > FIXED_LEN_BYTE_ARRAY.
> > > > > > - PARQUET-1510: Fix notEq for optional columns with null
> > > > values.
> > > > > > - PARQUET-1533: TestSnappy() throws OOM exception with
> > > > > Parquet-1485 change
> > > > > > - PARQUET-1531: Page row count limit causes empty pages
> to be
> > > > > written from
> > > > > > MessageColumnIO
> > > > > > - PARQUET-1544: Possible over-shading of modules
> > > > > >
> > > > > > The following change has been reverted so it is not part
> of
> > > any
> > > > > public
> > > > > > release:
> > > > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > > > > >
> > > > > > Please download, verify, and test. The vote will be open
> for
> > > at
> > > > > least 72
> > > > > > hours.
> > > > > >
> > > > > > Thanks,
> > > > > > Gabor
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > Ryan Blue
> > > > Software Engineer
> > > > Netflix
> > > >
> > >
> > >
> > > --
> > > Thanks & Best Regards
> > >
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Wes McKinney <we...@gmail.com>.
-1
Gabor's PGP key is unsigned.
$ gpg --verify apache-parquet-1.11.0.tar.gz.asc
gpg: assuming signed data in 'apache-parquet-1.11.0.tar.gz'
gpg: Signature made Tue 19 Mar 2019 08:55:48 AM CDT
gpg: using RSA key 6FB82970311551C7CEF131F5021057DBF048F543
gpg: Good signature from "Gabor Szadovszky <ga...@apache.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6FB8 2970 3115 51C7 CEF1 31F5 0210 57DB F048 F543
On Tue, Apr 16, 2019 at 4:10 AM Gabor Szadovszky <ga...@apache.org> wrote:
>
> Based on our release process (
> http://parquet.apache.org/documentation/how-to-release/) and the related
> scripts we use the final tag for an RC. So, the existence of this tag does
> not mean 1.11.0 is released.
> However, I agree this is misleading and not a good practice to remove
> already committed tags and re-add them to another place (when a new RC
> comes out). I think, we should update our release process to use RC tags
> and put the final tag only after it is officially released. But it is the
> story of the next release...
>
>
> On Sat, Apr 13, 2019 at 8:00 PM 俊杰陈 <cj...@gmail.com> wrote:
>
> > From the github release page, I see the 1.11.0 already released. Is it
> > still a rc version?
> > https://github.com/apache/parquet-mr/releases/tag/apache-parquet-1.11.0
> >
> > On Fri, Apr 12, 2019 at 8:10 AM Ryan Blue <rb...@netflix.com.invalid>
> > wrote:
> >
> > > Personally, I haven't had enough time to devote to Parquet lately and
> > that
> > > means I haven't validated that this release's new features are okay to
> > > release. I'm hoping sometime in the next few weeks I'll be able to vote
> > on
> > > this.
> > >
> > > On Thu, Apr 11, 2019 at 1:23 PM Andy Grove <An...@rms.com> wrote:
> > >
> > > > I'm curious if there is any update on this vote? The thread seems
> > eerily
> > > > quiet.
> > > >
> > > > Thanks.
> > > >
> > > > On 4/3/19, 10:38 AM, "Andy Grove" <An...@rms.com> wrote:
> > > >
> > > > CAUTION – UNVERIFIED EXTERNAL EMAIL
> > > >
> > > >
> > > > I have been able to run mvn verify and have also tested this RC
> > > > against our internal systems, with no issue.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > I have raised the issue about Hadoop-lzo, but that is present in
> > the
> > > > 1.10.1 release also.
> > > >
> > > > Andy.
> > > >
> > > >
> > > > On 3/20/19, 7:50 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID>
> > > wrote:
> > > >
> > > > CAUTION – UNVERIFIED EXTERNAL EMAIL
> > > >
> > > >
> > > > +1 (binding)
> > > >
> > > > signature matches
> > > > git hash matches the git tag
> > > > source tarball matches the git tag
> > > > unit tests and integration tests pass
> > > >
> > > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> > > gabor@apache.org>
> > > > wrote:
> > > >
> > > > > Dear Parquet Users and Developers,
> > > > >
> > > > > I propose the following RC to be released as the official
> > > Apache
> > > > > Parquet 1.11.0 release:
> > > > >
> > > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > > *
> > > > >
> > > > >
> > > >
> > >
> > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=v6kHzIIpJQp%2Fq7fuR%2ByHVwGV7vZ7lUKupyqKZwmQeFI%3D&reserved=0
> > > > >
> > > > > The release tarball, signature, and checksums are here:
> > > > > *
> > > > >
> > > >
> > >
> > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=RVlztCju4ZoZz5vnF8f5RxE7kPmZoKMj3Ipo4x0Aj4k%3D&reserved=0
> > > > >
> > > > > You can find the KEYS file here:
> > > > > *
> > > >
> > >
> > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=8xPAIJ4EkJPXXxZ2hTH%2BuJOtCOrCspYXkjsl%2B44Jb20%3D&reserved=0
> > > > >
> > > > > Binary artifacts are staged in Nexus here:
> > > > > *
> > > > >
> > > > >
> > > >
> > >
> > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342868310&sdata=%2FIW9qYFnwvuL7QgkrYxX%2BZWJ1fcaZz%2Bq1tRJWKfQERU%3D&reserved=0
> > > > >
> > > > > This release includes the following new features:
> > > > > - PARQUET-1201 - Column indexes
> > > > > - PARQUET-1253 - Support for new logical type representation
> > > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > > > - PARQUET-1388 - Nanosecond precision time and timestamp -
> > > > parquet-mr
> > > > >
> > > > > The release also includes bug fixes, including:
> > > > > - PARQUET-1472: Dictionary filter fails on
> > > FIXED_LEN_BYTE_ARRAY.
> > > > > - PARQUET-1510: Fix notEq for optional columns with null
> > > values.
> > > > > - PARQUET-1533: TestSnappy() throws OOM exception with
> > > > Parquet-1485 change
> > > > > - PARQUET-1531: Page row count limit causes empty pages to be
> > > > written from
> > > > > MessageColumnIO
> > > > > - PARQUET-1544: Possible over-shading of modules
> > > > >
> > > > > The following change has been reverted so it is not part of
> > any
> > > > public
> > > > > release:
> > > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > > > >
> > > > > Please download, verify, and test. The vote will be open for
> > at
> > > > least 72
> > > > > hours.
> > > > >
> > > > > Thanks,
> > > > > Gabor
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > > --
> > > Ryan Blue
> > > Software Engineer
> > > Netflix
> > >
> >
> >
> > --
> > Thanks & Best Regards
> >
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Gabor Szadovszky <ga...@apache.org>.
Based on our release process (
http://parquet.apache.org/documentation/how-to-release/) and the related
scripts we use the final tag for an RC. So, the existence of this tag does
not mean 1.11.0 is released.
However, I agree this is misleading and not a good practice to remove
already committed tags and re-add them to another place (when a new RC
comes out). I think, we should update our release process to use RC tags
and put the final tag only after it is officially released. But it is the
story of the next release...
On Sat, Apr 13, 2019 at 8:00 PM 俊杰陈 <cj...@gmail.com> wrote:
> From the github release page, I see the 1.11.0 already released. Is it
> still a rc version?
> https://github.com/apache/parquet-mr/releases/tag/apache-parquet-1.11.0
>
> On Fri, Apr 12, 2019 at 8:10 AM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
> > Personally, I haven't had enough time to devote to Parquet lately and
> that
> > means I haven't validated that this release's new features are okay to
> > release. I'm hoping sometime in the next few weeks I'll be able to vote
> on
> > this.
> >
> > On Thu, Apr 11, 2019 at 1:23 PM Andy Grove <An...@rms.com> wrote:
> >
> > > I'm curious if there is any update on this vote? The thread seems
> eerily
> > > quiet.
> > >
> > > Thanks.
> > >
> > > On 4/3/19, 10:38 AM, "Andy Grove" <An...@rms.com> wrote:
> > >
> > > CAUTION – UNVERIFIED EXTERNAL EMAIL
> > >
> > >
> > > I have been able to run mvn verify and have also tested this RC
> > > against our internal systems, with no issue.
> > >
> > > +1 (non-binding)
> > >
> > > I have raised the issue about Hadoop-lzo, but that is present in
> the
> > > 1.10.1 release also.
> > >
> > > Andy.
> > >
> > >
> > > On 3/20/19, 7:50 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID>
> > wrote:
> > >
> > > CAUTION – UNVERIFIED EXTERNAL EMAIL
> > >
> > >
> > > +1 (binding)
> > >
> > > signature matches
> > > git hash matches the git tag
> > > source tarball matches the git tag
> > > unit tests and integration tests pass
> > >
> > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> > gabor@apache.org>
> > > wrote:
> > >
> > > > Dear Parquet Users and Developers,
> > > >
> > > > I propose the following RC to be released as the official
> > Apache
> > > > Parquet 1.11.0 release:
> > > >
> > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > *
> > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=v6kHzIIpJQp%2Fq7fuR%2ByHVwGV7vZ7lUKupyqKZwmQeFI%3D&reserved=0
> > > >
> > > > The release tarball, signature, and checksums are here:
> > > > *
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=RVlztCju4ZoZz5vnF8f5RxE7kPmZoKMj3Ipo4x0Aj4k%3D&reserved=0
> > > >
> > > > You can find the KEYS file here:
> > > > *
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=8xPAIJ4EkJPXXxZ2hTH%2BuJOtCOrCspYXkjsl%2B44Jb20%3D&reserved=0
> > > >
> > > > Binary artifacts are staged in Nexus here:
> > > > *
> > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342868310&sdata=%2FIW9qYFnwvuL7QgkrYxX%2BZWJ1fcaZz%2Bq1tRJWKfQERU%3D&reserved=0
> > > >
> > > > This release includes the following new features:
> > > > - PARQUET-1201 - Column indexes
> > > > - PARQUET-1253 - Support for new logical type representation
> > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > > - PARQUET-1388 - Nanosecond precision time and timestamp -
> > > parquet-mr
> > > >
> > > > The release also includes bug fixes, including:
> > > > - PARQUET-1472: Dictionary filter fails on
> > FIXED_LEN_BYTE_ARRAY.
> > > > - PARQUET-1510: Fix notEq for optional columns with null
> > values.
> > > > - PARQUET-1533: TestSnappy() throws OOM exception with
> > > Parquet-1485 change
> > > > - PARQUET-1531: Page row count limit causes empty pages to be
> > > written from
> > > > MessageColumnIO
> > > > - PARQUET-1544: Possible over-shading of modules
> > > >
> > > > The following change has been reverted so it is not part of
> any
> > > public
> > > > release:
> > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > > >
> > > > Please download, verify, and test. The vote will be open for
> at
> > > least 72
> > > > hours.
> > > >
> > > > Thanks,
> > > > Gabor
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
> >
>
>
> --
> Thanks & Best Regards
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by 俊杰陈 <cj...@gmail.com>.
From the github release page, I see the 1.11.0 already released. Is it
still a rc version?
https://github.com/apache/parquet-mr/releases/tag/apache-parquet-1.11.0
On Fri, Apr 12, 2019 at 8:10 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
> Personally, I haven't had enough time to devote to Parquet lately and that
> means I haven't validated that this release's new features are okay to
> release. I'm hoping sometime in the next few weeks I'll be able to vote on
> this.
>
> On Thu, Apr 11, 2019 at 1:23 PM Andy Grove <An...@rms.com> wrote:
>
> > I'm curious if there is any update on this vote? The thread seems eerily
> > quiet.
> >
> > Thanks.
> >
> > On 4/3/19, 10:38 AM, "Andy Grove" <An...@rms.com> wrote:
> >
> > CAUTION – UNVERIFIED EXTERNAL EMAIL
> >
> >
> > I have been able to run mvn verify and have also tested this RC
> > against our internal systems, with no issue.
> >
> > +1 (non-binding)
> >
> > I have raised the issue about Hadoop-lzo, but that is present in the
> > 1.10.1 release also.
> >
> > Andy.
> >
> >
> > On 3/20/19, 7:50 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID>
> wrote:
> >
> > CAUTION – UNVERIFIED EXTERNAL EMAIL
> >
> >
> > +1 (binding)
> >
> > signature matches
> > git hash matches the git tag
> > source tarball matches the git tag
> > unit tests and integration tests pass
> >
> > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> gabor@apache.org>
> > wrote:
> >
> > > Dear Parquet Users and Developers,
> > >
> > > I propose the following RC to be released as the official
> Apache
> > > Parquet 1.11.0 release:
> > >
> > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > * This corresponds to the tag: apache-parquet-1.11.0
> > > *
> > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=v6kHzIIpJQp%2Fq7fuR%2ByHVwGV7vZ7lUKupyqKZwmQeFI%3D&reserved=0
> > >
> > > The release tarball, signature, and checksums are here:
> > > *
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=RVlztCju4ZoZz5vnF8f5RxE7kPmZoKMj3Ipo4x0Aj4k%3D&reserved=0
> > >
> > > You can find the KEYS file here:
> > > *
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=8xPAIJ4EkJPXXxZ2hTH%2BuJOtCOrCspYXkjsl%2B44Jb20%3D&reserved=0
> > >
> > > Binary artifacts are staged in Nexus here:
> > > *
> > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342868310&sdata=%2FIW9qYFnwvuL7QgkrYxX%2BZWJ1fcaZz%2Bq1tRJWKfQERU%3D&reserved=0
> > >
> > > This release includes the following new features:
> > > - PARQUET-1201 - Column indexes
> > > - PARQUET-1253 - Support for new logical type representation
> > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > - PARQUET-1388 - Nanosecond precision time and timestamp -
> > parquet-mr
> > >
> > > The release also includes bug fixes, including:
> > > - PARQUET-1472: Dictionary filter fails on
> FIXED_LEN_BYTE_ARRAY.
> > > - PARQUET-1510: Fix notEq for optional columns with null
> values.
> > > - PARQUET-1533: TestSnappy() throws OOM exception with
> > Parquet-1485 change
> > > - PARQUET-1531: Page row count limit causes empty pages to be
> > written from
> > > MessageColumnIO
> > > - PARQUET-1544: Possible over-shading of modules
> > >
> > > The following change has been reverted so it is not part of any
> > public
> > > release:
> > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > >
> > > Please download, verify, and test. The vote will be open for at
> > least 72
> > > hours.
> > >
> > > Thanks,
> > > Gabor
> > >
> >
> >
> >
> >
> >
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
--
Thanks & Best Regards
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
Personally, I haven't had enough time to devote to Parquet lately and that
means I haven't validated that this release's new features are okay to
release. I'm hoping sometime in the next few weeks I'll be able to vote on
this.
On Thu, Apr 11, 2019 at 1:23 PM Andy Grove <An...@rms.com> wrote:
> I'm curious if there is any update on this vote? The thread seems eerily
> quiet.
>
> Thanks.
>
> On 4/3/19, 10:38 AM, "Andy Grove" <An...@rms.com> wrote:
>
> CAUTION – UNVERIFIED EXTERNAL EMAIL
>
>
> I have been able to run mvn verify and have also tested this RC
> against our internal systems, with no issue.
>
> +1 (non-binding)
>
> I have raised the issue about Hadoop-lzo, but that is present in the
> 1.10.1 release also.
>
> Andy.
>
>
> On 3/20/19, 7:50 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
>
> CAUTION – UNVERIFIED EXTERNAL EMAIL
>
>
> +1 (binding)
>
> signature matches
> git hash matches the git tag
> source tarball matches the git tag
> unit tests and integration tests pass
>
> On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <ga...@apache.org>
> wrote:
>
> > Dear Parquet Users and Developers,
> >
> > I propose the following RC to be released as the official Apache
> > Parquet 1.11.0 release:
> >
> > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > * This corresponds to the tag: apache-parquet-1.11.0
> > *
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=v6kHzIIpJQp%2Fq7fuR%2ByHVwGV7vZ7lUKupyqKZwmQeFI%3D&reserved=0
> >
> > The release tarball, signature, and checksums are here:
> > *
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=RVlztCju4ZoZz5vnF8f5RxE7kPmZoKMj3Ipo4x0Aj4k%3D&reserved=0
> >
> > You can find the KEYS file here:
> > *
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=8xPAIJ4EkJPXXxZ2hTH%2BuJOtCOrCspYXkjsl%2B44Jb20%3D&reserved=0
> >
> > Binary artifacts are staged in Nexus here:
> > *
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342868310&sdata=%2FIW9qYFnwvuL7QgkrYxX%2BZWJ1fcaZz%2Bq1tRJWKfQERU%3D&reserved=0
> >
> > This release includes the following new features:
> > - PARQUET-1201 - Column indexes
> > - PARQUET-1253 - Support for new logical type representation
> > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > - PARQUET-1388 - Nanosecond precision time and timestamp -
> parquet-mr
> >
> > The release also includes bug fixes, including:
> > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> > - PARQUET-1510: Fix notEq for optional columns with null values.
> > - PARQUET-1533: TestSnappy() throws OOM exception with
> Parquet-1485 change
> > - PARQUET-1531: Page row count limit causes empty pages to be
> written from
> > MessageColumnIO
> > - PARQUET-1544: Possible over-shading of modules
> >
> > The following change has been reverted so it is not part of any
> public
> > release:
> > - PARQUET-1381: Add merge blocks command to parquet-tools
> >
> > Please download, verify, and test. The vote will be open for at
> least 72
> > hours.
> >
> > Thanks,
> > Gabor
> >
>
>
>
>
>
--
Ryan Blue
Software Engineer
Netflix
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Andy Grove <An...@rms.com>.
I'm curious if there is any update on this vote? The thread seems eerily quiet.
Thanks.
On 4/3/19, 10:38 AM, "Andy Grove" <An...@rms.com> wrote:
CAUTION – UNVERIFIED EXTERNAL EMAIL
I have been able to run mvn verify and have also tested this RC against our internal systems, with no issue.
+1 (non-binding)
I have raised the issue about Hadoop-lzo, but that is present in the 1.10.1 release also.
Andy.
On 3/20/19, 7:50 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
CAUTION – UNVERIFIED EXTERNAL EMAIL
+1 (binding)
signature matches
git hash matches the git tag
source tarball matches the git tag
unit tests and integration tests pass
On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <ga...@apache.org> wrote:
> Dear Parquet Users and Developers,
>
> I propose the following RC to be released as the official Apache
> Parquet 1.11.0 release:
>
> The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> * This corresponds to the tag: apache-parquet-1.11.0
> *
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=v6kHzIIpJQp%2Fq7fuR%2ByHVwGV7vZ7lUKupyqKZwmQeFI%3D&reserved=0
>
> The release tarball, signature, and checksums are here:
> *
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=RVlztCju4ZoZz5vnF8f5RxE7kPmZoKMj3Ipo4x0Aj4k%3D&reserved=0
>
> You can find the KEYS file here:
> * https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342858310&sdata=8xPAIJ4EkJPXXxZ2hTH%2BuJOtCOrCspYXkjsl%2B44Jb20%3D&reserved=0
>
> Binary artifacts are staged in Nexus here:
> *
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cc45463142cfe401f12b708d6b852dac3%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899063342868310&sdata=%2FIW9qYFnwvuL7QgkrYxX%2BZWJ1fcaZz%2Bq1tRJWKfQERU%3D&reserved=0
>
> This release includes the following new features:
> - PARQUET-1201 - Column indexes
> - PARQUET-1253 - Support for new logical type representation
> - PARQUET-1381 - Add merge blocks command to parquet-tools
> - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
>
> The release also includes bug fixes, including:
> - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> - PARQUET-1510: Fix notEq for optional columns with null values.
> - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485 change
> - PARQUET-1531: Page row count limit causes empty pages to be written from
> MessageColumnIO
> - PARQUET-1544: Possible over-shading of modules
>
> The following change has been reverted so it is not part of any public
> release:
> - PARQUET-1381: Add merge blocks command to parquet-tools
>
> Please download, verify, and test. The vote will be open for at least 72
> hours.
>
> Thanks,
> Gabor
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Andy Grove <An...@rms.com>.
I have been able to run mvn verify and have also tested this RC against our internal systems, with no issue.
+1 (non-binding)
I have raised the issue about Hadoop-lzo, but that is present in the 1.10.1 release also.
Andy.
On 3/20/19, 7:50 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
CAUTION – UNVERIFIED EXTERNAL EMAIL
+1 (binding)
signature matches
git hash matches the git tag
source tarball matches the git tag
unit tests and integration tests pass
On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <ga...@apache.org> wrote:
> Dear Parquet Users and Developers,
>
> I propose the following RC to be released as the official Apache
> Parquet 1.11.0 release:
>
> The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> * This corresponds to the tag: apache-parquet-1.11.0
> *
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7Cf8203f53dd0941c8dc6508d6ad3b0906%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636886866421222421&sdata=5gRop5pefD3k7uKdAXfF%2Fy%2Bg2YoCsfU1lsEq6TwMJwA%3D&reserved=0
>
> The release tarball, signature, and checksums are here:
> *
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cf8203f53dd0941c8dc6508d6ad3b0906%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636886866421232430&sdata=p014KEZuk8XS1FMmPfVU2TsiwEnJDWpNoRfKFwTwmTY%3D&reserved=0
>
> You can find the KEYS file here:
> * https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7Cf8203f53dd0941c8dc6508d6ad3b0906%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636886866421232430&sdata=rS6iPSQPPKZNmv4Dy9x07i1rykYOi5YtLU3PZE1kGzQ%3D&reserved=0
>
> Binary artifacts are staged in Nexus here:
> *
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7Cf8203f53dd0941c8dc6508d6ad3b0906%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636886866421232430&sdata=aYscjUdVzTkpQ0wXasOU%2Fj%2B%2F9GbjR1zOeaoOPXQkj3c%3D&reserved=0
>
> This release includes the following new features:
> - PARQUET-1201 - Column indexes
> - PARQUET-1253 - Support for new logical type representation
> - PARQUET-1381 - Add merge blocks command to parquet-tools
> - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
>
> The release also includes bug fixes, including:
> - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> - PARQUET-1510: Fix notEq for optional columns with null values.
> - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485 change
> - PARQUET-1531: Page row count limit causes empty pages to be written from
> MessageColumnIO
> - PARQUET-1544: Possible over-shading of modules
>
> The following change has been reverted so it is not part of any public
> release:
> - PARQUET-1381: Add merge blocks command to parquet-tools
>
> Please download, verify, and test. The vote will be open for at least 72
> hours.
>
> Thanks,
> Gabor
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Zoltan Ivanfi <zi...@cloudera.com.INVALID>.
Hi,
I have made some experiments as well and this gets more and more
interesting. Let's move this conversation to the JIRA though (thanks for
creating it by the way).
Br,
Zoltan
On Wed, Apr 3, 2019 at 5:17 PM Andy Grove <An...@rms.com> wrote:
> It works after configuring the twitter Maven repo. Here is the JIRA:
> https://issues.apache.org/jira/browse/PARQUET-1556
>
> On 4/3/19, 9:07 AM, "Andy Grove" <An...@rms.com> wrote:
>
> CAUTION – UNVERIFIED EXTERNAL EMAIL
>
>
> This artifact isn't available in maven central. It is in Twitter's
> maven repo.
>
> I am working on setting up my m2 settings to be able to read from
> there, but this is something that really needs to be documented.
>
> Once I figure it out, I will create a JIRA + PR to update the README.
>
> Thanks.
>
> On 4/3/19, 8:55 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
>
> CAUTION – UNVERIFIED EXTERNAL EMAIL
>
>
> Hi,
>
> That error message is quite surprising, Maven should be able to
> find that
> artifact without any manual configuration. It may have been some
> temporary
> network issue with accessing the repository. Could you try it
> again? If it
> keeps failing, Maven may have cached the unavailability of that
> artifact,
> which can be cleaned by issuing
>
> mvn dependency:purge-local-repository
> -DmanualInclude=com.hadoop.gplcompression:hadoop-lzo
>
> In case you still continue to get the same failure, you may be
> behind a
> stale artifact cache or some other kind of proxy or that does not
> allow you
> to access this artifact directly.
>
> Br,
>
> Zoltan
>
>
> On Wed, Apr 3, 2019 at 2:14 PM XU Qinghui <qi...@gmail.com>
> wrote:
>
> > Hello, Andy
> >
> > I think this dependency is in maven repository:
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmvnrepository.com%2Fartifact%2Fcom.hadoop.gplcompression%2Fhadoop-lzo%2F0.4.16&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=Bpzt4O1bvNXqy%2FWO7KKYh2sECoU6KV94fZXvIZ5M0Tk%3D&reserved=0
> > Maybe you need to configure your `.m2/settings.xml` to have the
> repository,
> > here's some doc from maven:
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaven.apache.org%2Fguides%2Fmini%2Fguide-multiple-repositories.html&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=DwAmqo7IPC7EYNZE03b76EKYdzruQ%2FI6NFpPmLEELOo%3D&reserved=0
> >
> > Best wishes,
> >
> > Le mer. 3 avr. 2019 à 13:33, Andy Grove <An...@rms.com> a
> écrit :
> >
> > > I've been watching the RC process here and would like to help
> verify the
> > > release, since I have co-workers wanting to upgrade to 1.11
> > >
> > > When I follow the instructions to verify the release I am
> running into an
> > > issue resolving a dependency:
> > >
> > > Could not resolve dependencies for project
> > > org.apache.parquet:parquet-thrift:jar:1.11.0: Could not find
> artifact
> > > com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16
> > >
> > > I don't see anything in the README in the repo that explains
> how I should
> > > configure maven to find this dependency. I don’t even see a
> direct
> > > reference to this jar in maven, so would appreciate some
> guidance.
> > >
> > > Thanks,
> > >
> > > Andy,
> > >
> > >
> > > On 3/21/19, 3:40 PM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID>
> wrote:
> > >
> > > CAUTION – UNVERIFIED EXTERNAL EMAIL
> > >
> > >
> > > Hi Wes,
> > >
> > > Here is a list of steps (the first part is probably the
> same as for
> > > parquet-cpp):
> > >
> > > 1. Download the files from
> > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=09cNlF5%2FzvJwIqAT9tsixh82aoJEna6h1voR7C3S%2BXc%3D&reserved=0
> > > 2. Download the KEYS file from
> > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=Dad6aN4TzDgVmoRDrAxDvTxEO6qFyVYGkHlVvK6ez%2Bo%3D&reserved=0
> > > 3. Import the GPG keys by running `gpg --import KEYS`.
> > > 4. Verify the signature by running `gpg --verify
> > > apache-parquet-1.11.0.tar.gz.asc`. It should say "Good
> signature",
> > the
> > > warning about the key not being trusted can be ignored.
> > > 5. Extract the source code: `tar xzf
> apache-parquet-1.11.0.tar.gz`.
> > > 6. Change to the source directory: `cd
> apache-parquet-1.11.0`.
> > > 7. Make sure that you have a thrift compiler and that you
> have the
> > > correct
> > > version by issuing `thrift -version`. It should print
> "Thrift version
> > > 0.12.0". If you don't have it, you need to install it in a
> manner
> > that
> > > is
> > > specific to your system (for example, `sudo apt install
> > > thrift-compiler` or
> > > `sudo yum install thrift-compiler`). If your package
> manager only
> > > offers an
> > > older version, you may still use that with the workaround
> shown
> > below.
> > > 8. Test the release by issuing `mvn clean verify`. If you
> have an
> > older
> > > thrift compiler, for example 0.9.3, you have to issue `mvn
> > > -Dformat.thrift.version=0.9.3 -Dthrift.version=0.9.3 clean
> install`
> > > instead. This may or may not work depending on how old
> your thrift
> > > version
> > > is (>= 0.9.0 should work). Running this command can take
> around 15
> > > minutes
> > > and should print "BUILD SUCCESS" at the end (followed by a
> few extra
> > > lines).
> > >
> > > Br,
> > >
> > > Zoltan
> > >
> > > On Thu, Mar 21, 2019 at 7:58 PM Wes McKinney <
> wesmckinn@gmail.com>
> > > wrote:
> > >
> > > > Are there any instructions written down about how to
> verify this
> > > release?
> > > >
> > > > On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi
> > > <zi...@cloudera.com.invalid>
> > > > wrote:
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > signature matches
> > > > > git hash matches the git tag
> > > > > source tarball matches the git tag
> > > > > unit tests and integration tests pass
> > > > >
> > > > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> > gabor@apache.org
> > > >
> > > > wrote:
> > > > >
> > > > > > Dear Parquet Users and Developers,
> > > > > >
> > > > > > I propose the following RC to be released as the
> official
> > Apache
> > > > > > Parquet 1.11.0 release:
> > > > > >
> > > > > > The commit id is
> 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > > > *
> > > > > >
> > > > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=ZKYiD%2F%2FDwP7zDadpxBvtSF4T%2FjXCdwIsAT8w%2FBQcxAQ%3D&reserved=0
> > > > > >
> > > > > > The release tarball, signature, and checksums are
> here:
> > > > > > *
> > > > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=v9PAI9BCY0HTGld7Zlb%2F8gJYbQH%2Fy1WmOw87ZzWymUU%3D&reserved=0
> > > > > >
> > > > > > You can find the KEYS file here:
> > > > > > *
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fformatdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=VqkSY5roEqYKo%2B09o6Wh2NZQWKkdbAKDvtWIdbLjccE%3D&reserved=0
> > > > <
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=2VCbJrdrUE1g3dN3VUqN%2FAq1Gt6CcBAGmGfx3CjlauI%3D&reserved=0
> > > >
> > > > > >
> > > > > > Binary artifacts are staged in Nexus here:
> > > > > > *
> > > > > >
> > > > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=Ff9wOSvCIG25l9N89STbYJfod7Oo0cb9pWqfOlaRD9s%3D&reserved=0
> > > > > >
> > > > > > This release includes the following new features:
> > > > > > - PARQUET-1201 - Column indexes
> > > > > > - PARQUET-1253 - Support for new logical type
> representation
> > > > > > - PARQUET-1381 - Add merge blocks command to
> parquet-tools
> > > > > > - PARQUET-1388 - Nanosecond precision time and
> timestamp -
> > > parquet-mr
> > > > > >
> > > > > > The release also includes bug fixes, including:
> > > > > > - PARQUET-1472: Dictionary filter fails on
> > FIXED_LEN_BYTE_ARRAY.
> > > > > > - PARQUET-1510: Fix notEq for optional columns with
> null
> > values.
> > > > > > - PARQUET-1533: TestSnappy() throws OOM exception
> with
> > > Parquet-1485
> > > > change
> > > > > > - PARQUET-1531: Page row count limit causes empty
> pages to be
> > > written
> > > > from
> > > > > > MessageColumnIO
> > > > > > - PARQUET-1544: Possible over-shading of modules
> > > > > >
> > > > > > The following change has been reverted so it is not
> part of any
> > > public
> > > > > > release:
> > > > > > - PARQUET-1381: Add merge blocks command to
> parquet-tools
> > > > > >
> > > > > > Please download, verify, and test. The vote will be
> open for at
> > > least
> > > > 72
> > > > > > hours.
> > > > > >
> > > > > > Thanks,
> > > > > > Gabor
> > > > > >
> > > >
> > >
> > >
> > >
> >
>
>
>
>
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Andy Grove <An...@rms.com>.
It works after configuring the twitter Maven repo. Here is the JIRA: https://issues.apache.org/jira/browse/PARQUET-1556
On 4/3/19, 9:07 AM, "Andy Grove" <An...@rms.com> wrote:
CAUTION – UNVERIFIED EXTERNAL EMAIL
This artifact isn't available in maven central. It is in Twitter's maven repo.
I am working on setting up my m2 settings to be able to read from there, but this is something that really needs to be documented.
Once I figure it out, I will create a JIRA + PR to update the README.
Thanks.
On 4/3/19, 8:55 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
CAUTION – UNVERIFIED EXTERNAL EMAIL
Hi,
That error message is quite surprising, Maven should be able to find that
artifact without any manual configuration. It may have been some temporary
network issue with accessing the repository. Could you try it again? If it
keeps failing, Maven may have cached the unavailability of that artifact,
which can be cleaned by issuing
mvn dependency:purge-local-repository
-DmanualInclude=com.hadoop.gplcompression:hadoop-lzo
In case you still continue to get the same failure, you may be behind a
stale artifact cache or some other kind of proxy or that does not allow you
to access this artifact directly.
Br,
Zoltan
On Wed, Apr 3, 2019 at 2:14 PM XU Qinghui <qi...@gmail.com> wrote:
> Hello, Andy
>
> I think this dependency is in maven repository:
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmvnrepository.com%2Fartifact%2Fcom.hadoop.gplcompression%2Fhadoop-lzo%2F0.4.16&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=Bpzt4O1bvNXqy%2FWO7KKYh2sECoU6KV94fZXvIZ5M0Tk%3D&reserved=0
> Maybe you need to configure your `.m2/settings.xml` to have the repository,
> here's some doc from maven:
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaven.apache.org%2Fguides%2Fmini%2Fguide-multiple-repositories.html&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=DwAmqo7IPC7EYNZE03b76EKYdzruQ%2FI6NFpPmLEELOo%3D&reserved=0
>
> Best wishes,
>
> Le mer. 3 avr. 2019 à 13:33, Andy Grove <An...@rms.com> a écrit :
>
> > I've been watching the RC process here and would like to help verify the
> > release, since I have co-workers wanting to upgrade to 1.11
> >
> > When I follow the instructions to verify the release I am running into an
> > issue resolving a dependency:
> >
> > Could not resolve dependencies for project
> > org.apache.parquet:parquet-thrift:jar:1.11.0: Could not find artifact
> > com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16
> >
> > I don't see anything in the README in the repo that explains how I should
> > configure maven to find this dependency. I don’t even see a direct
> > reference to this jar in maven, so would appreciate some guidance.
> >
> > Thanks,
> >
> > Andy,
> >
> >
> > On 3/21/19, 3:40 PM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
> >
> > CAUTION – UNVERIFIED EXTERNAL EMAIL
> >
> >
> > Hi Wes,
> >
> > Here is a list of steps (the first part is probably the same as for
> > parquet-cpp):
> >
> > 1. Download the files from
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=09cNlF5%2FzvJwIqAT9tsixh82aoJEna6h1voR7C3S%2BXc%3D&reserved=0
> > 2. Download the KEYS file from
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=Dad6aN4TzDgVmoRDrAxDvTxEO6qFyVYGkHlVvK6ez%2Bo%3D&reserved=0
> > 3. Import the GPG keys by running `gpg --import KEYS`.
> > 4. Verify the signature by running `gpg --verify
> > apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature",
> the
> > warning about the key not being trusted can be ignored.
> > 5. Extract the source code: `tar xzf apache-parquet-1.11.0.tar.gz`.
> > 6. Change to the source directory: `cd apache-parquet-1.11.0`.
> > 7. Make sure that you have a thrift compiler and that you have the
> > correct
> > version by issuing `thrift -version`. It should print "Thrift version
> > 0.12.0". If you don't have it, you need to install it in a manner
> that
> > is
> > specific to your system (for example, `sudo apt install
> > thrift-compiler` or
> > `sudo yum install thrift-compiler`). If your package manager only
> > offers an
> > older version, you may still use that with the workaround shown
> below.
> > 8. Test the release by issuing `mvn clean verify`. If you have an
> older
> > thrift compiler, for example 0.9.3, you have to issue `mvn
> > -Dformat.thrift.version=0.9.3 -Dthrift.version=0.9.3 clean install`
> > instead. This may or may not work depending on how old your thrift
> > version
> > is (>= 0.9.0 should work). Running this command can take around 15
> > minutes
> > and should print "BUILD SUCCESS" at the end (followed by a few extra
> > lines).
> >
> > Br,
> >
> > Zoltan
> >
> > On Thu, Mar 21, 2019 at 7:58 PM Wes McKinney <we...@gmail.com>
> > wrote:
> >
> > > Are there any instructions written down about how to verify this
> > release?
> > >
> > > On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi
> > <zi...@cloudera.com.invalid>
> > > wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > > signature matches
> > > > git hash matches the git tag
> > > > source tarball matches the git tag
> > > > unit tests and integration tests pass
> > > >
> > > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> gabor@apache.org
> > >
> > > wrote:
> > > >
> > > > > Dear Parquet Users and Developers,
> > > > >
> > > > > I propose the following RC to be released as the official
> Apache
> > > > > Parquet 1.11.0 release:
> > > > >
> > > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > > *
> > > > >
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=ZKYiD%2F%2FDwP7zDadpxBvtSF4T%2FjXCdwIsAT8w%2FBQcxAQ%3D&reserved=0
> > > > >
> > > > > The release tarball, signature, and checksums are here:
> > > > > *
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=v9PAI9BCY0HTGld7Zlb%2F8gJYbQH%2Fy1WmOw87ZzWymUU%3D&reserved=0
> > > > >
> > > > > You can find the KEYS file here:
> > > > > *
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fformatdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=VqkSY5roEqYKo%2B09o6Wh2NZQWKkdbAKDvtWIdbLjccE%3D&reserved=0
> > > <
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=2VCbJrdrUE1g3dN3VUqN%2FAq1Gt6CcBAGmGfx3CjlauI%3D&reserved=0
> > >
> > > > >
> > > > > Binary artifacts are staged in Nexus here:
> > > > > *
> > > > >
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=Ff9wOSvCIG25l9N89STbYJfod7Oo0cb9pWqfOlaRD9s%3D&reserved=0
> > > > >
> > > > > This release includes the following new features:
> > > > > - PARQUET-1201 - Column indexes
> > > > > - PARQUET-1253 - Support for new logical type representation
> > > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > > > - PARQUET-1388 - Nanosecond precision time and timestamp -
> > parquet-mr
> > > > >
> > > > > The release also includes bug fixes, including:
> > > > > - PARQUET-1472: Dictionary filter fails on
> FIXED_LEN_BYTE_ARRAY.
> > > > > - PARQUET-1510: Fix notEq for optional columns with null
> values.
> > > > > - PARQUET-1533: TestSnappy() throws OOM exception with
> > Parquet-1485
> > > change
> > > > > - PARQUET-1531: Page row count limit causes empty pages to be
> > written
> > > from
> > > > > MessageColumnIO
> > > > > - PARQUET-1544: Possible over-shading of modules
> > > > >
> > > > > The following change has been reverted so it is not part of any
> > public
> > > > > release:
> > > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > > > >
> > > > > Please download, verify, and test. The vote will be open for at
> > least
> > > 72
> > > > > hours.
> > > > >
> > > > > Thanks,
> > > > > Gabor
> > > > >
> > >
> >
> >
> >
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Andy Grove <An...@rms.com>.
This artifact isn't available in maven central. It is in Twitter's maven repo.
I am working on setting up my m2 settings to be able to read from there, but this is something that really needs to be documented.
Once I figure it out, I will create a JIRA + PR to update the README.
Thanks.
On 4/3/19, 8:55 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
CAUTION – UNVERIFIED EXTERNAL EMAIL
Hi,
That error message is quite surprising, Maven should be able to find that
artifact without any manual configuration. It may have been some temporary
network issue with accessing the repository. Could you try it again? If it
keeps failing, Maven may have cached the unavailability of that artifact,
which can be cleaned by issuing
mvn dependency:purge-local-repository
-DmanualInclude=com.hadoop.gplcompression:hadoop-lzo
In case you still continue to get the same failure, you may be behind a
stale artifact cache or some other kind of proxy or that does not allow you
to access this artifact directly.
Br,
Zoltan
On Wed, Apr 3, 2019 at 2:14 PM XU Qinghui <qi...@gmail.com> wrote:
> Hello, Andy
>
> I think this dependency is in maven repository:
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmvnrepository.com%2Fartifact%2Fcom.hadoop.gplcompression%2Fhadoop-lzo%2F0.4.16&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001407997784&sdata=RpIKEzypbGLJFh0%2FgqDBKlkYoMTS6SJPQer95PszzZA%3D&reserved=0
> Maybe you need to configure your `.m2/settings.xml` to have the repository,
> here's some doc from maven:
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaven.apache.org%2Fguides%2Fmini%2Fguide-multiple-repositories.html&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001407997784&sdata=PEQr6hvv1dnNi5p6WAtU5o0xjvx3X9CqnTTV0ofVsjg%3D&reserved=0
>
> Best wishes,
>
> Le mer. 3 avr. 2019 à 13:33, Andy Grove <An...@rms.com> a écrit :
>
> > I've been watching the RC process here and would like to help verify the
> > release, since I have co-workers wanting to upgrade to 1.11
> >
> > When I follow the instructions to verify the release I am running into an
> > issue resolving a dependency:
> >
> > Could not resolve dependencies for project
> > org.apache.parquet:parquet-thrift:jar:1.11.0: Could not find artifact
> > com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16
> >
> > I don't see anything in the README in the repo that explains how I should
> > configure maven to find this dependency. I don’t even see a direct
> > reference to this jar in maven, so would appreciate some guidance.
> >
> > Thanks,
> >
> > Andy,
> >
> >
> > On 3/21/19, 3:40 PM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
> >
> > CAUTION – UNVERIFIED EXTERNAL EMAIL
> >
> >
> > Hi Wes,
> >
> > Here is a list of steps (the first part is probably the same as for
> > parquet-cpp):
> >
> > 1. Download the files from
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001407997784&sdata=7PYUYgwmDw%2BqJplXViaytfw2%2FQQOh00TN7FEHGKhJtw%3D&reserved=0
> > 2. Download the KEYS file from
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001407997784&sdata=drV7YZLYJOsaK1ta%2Bfu4gguxmvGP20YzFbJVVdXtF0M%3D&reserved=0
> > 3. Import the GPG keys by running `gpg --import KEYS`.
> > 4. Verify the signature by running `gpg --verify
> > apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature",
> the
> > warning about the key not being trusted can be ignored.
> > 5. Extract the source code: `tar xzf apache-parquet-1.11.0.tar.gz`.
> > 6. Change to the source directory: `cd apache-parquet-1.11.0`.
> > 7. Make sure that you have a thrift compiler and that you have the
> > correct
> > version by issuing `thrift -version`. It should print "Thrift version
> > 0.12.0". If you don't have it, you need to install it in a manner
> that
> > is
> > specific to your system (for example, `sudo apt install
> > thrift-compiler` or
> > `sudo yum install thrift-compiler`). If your package manager only
> > offers an
> > older version, you may still use that with the workaround shown
> below.
> > 8. Test the release by issuing `mvn clean verify`. If you have an
> older
> > thrift compiler, for example 0.9.3, you have to issue `mvn
> > -Dformat.thrift.version=0.9.3 -Dthrift.version=0.9.3 clean install`
> > instead. This may or may not work depending on how old your thrift
> > version
> > is (>= 0.9.0 should work). Running this command can take around 15
> > minutes
> > and should print "BUILD SUCCESS" at the end (followed by a few extra
> > lines).
> >
> > Br,
> >
> > Zoltan
> >
> > On Thu, Mar 21, 2019 at 7:58 PM Wes McKinney <we...@gmail.com>
> > wrote:
> >
> > > Are there any instructions written down about how to verify this
> > release?
> > >
> > > On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi
> > <zi...@cloudera.com.invalid>
> > > wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > > signature matches
> > > > git hash matches the git tag
> > > > source tarball matches the git tag
> > > > unit tests and integration tests pass
> > > >
> > > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> gabor@apache.org
> > >
> > > wrote:
> > > >
> > > > > Dear Parquet Users and Developers,
> > > > >
> > > > > I propose the following RC to be released as the official
> Apache
> > > > > Parquet 1.11.0 release:
> > > > >
> > > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > > *
> > > > >
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001408007793&sdata=oFkElFs3tclPAJwoxB1nBv%2FlNcLFxHIYPQVFtOdikGc%3D&reserved=0
> > > > >
> > > > > The release tarball, signature, and checksums are here:
> > > > > *
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001408007793&sdata=hbTVHQfazBMEsNKJSs2tk0nUlSTJ800rZgzwjC8tTbk%3D&reserved=0
> > > > >
> > > > > You can find the KEYS file here:
> > > > > *
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fformatdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001408007793&sdata=peJjszgj7vBJpxTn%2BglNiuAgRf2bL0%2FK0Md5gLl8A7s%3D&reserved=0
> > > <
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001408007793&sdata=GxLCnU0NjoyXZ3XJCRI0QPlJyUTyus814k%2FironJ7PA%3D&reserved=0
> > >
> > > > >
> > > > > Binary artifacts are staged in Nexus here:
> > > > > *
> > > > >
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001408007793&sdata=HJZSphBPPRw81jLJ%2FqBHfT4TlqAsGBhCNGWQIr8LNlE%3D&reserved=0
> > > > >
> > > > > This release includes the following new features:
> > > > > - PARQUET-1201 - Column indexes
> > > > > - PARQUET-1253 - Support for new logical type representation
> > > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > > > - PARQUET-1388 - Nanosecond precision time and timestamp -
> > parquet-mr
> > > > >
> > > > > The release also includes bug fixes, including:
> > > > > - PARQUET-1472: Dictionary filter fails on
> FIXED_LEN_BYTE_ARRAY.
> > > > > - PARQUET-1510: Fix notEq for optional columns with null
> values.
> > > > > - PARQUET-1533: TestSnappy() throws OOM exception with
> > Parquet-1485
> > > change
> > > > > - PARQUET-1531: Page row count limit causes empty pages to be
> > written
> > > from
> > > > > MessageColumnIO
> > > > > - PARQUET-1544: Possible over-shading of modules
> > > > >
> > > > > The following change has been reverted so it is not part of any
> > public
> > > > > release:
> > > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > > > >
> > > > > Please download, verify, and test. The vote will be open for at
> > least
> > > 72
> > > > > hours.
> > > > >
> > > > > Thanks,
> > > > > Gabor
> > > > >
> > >
> >
> >
> >
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Zoltan Ivanfi <zi...@cloudera.com.INVALID>.
Hi,
That error message is quite surprising, Maven should be able to find that
artifact without any manual configuration. It may have been some temporary
network issue with accessing the repository. Could you try it again? If it
keeps failing, Maven may have cached the unavailability of that artifact,
which can be cleaned by issuing
mvn dependency:purge-local-repository
-DmanualInclude=com.hadoop.gplcompression:hadoop-lzo
In case you still continue to get the same failure, you may be behind a
stale artifact cache or some other kind of proxy or that does not allow you
to access this artifact directly.
Br,
Zoltan
On Wed, Apr 3, 2019 at 2:14 PM XU Qinghui <qi...@gmail.com> wrote:
> Hello, Andy
>
> I think this dependency is in maven repository:
>
> https://mvnrepository.com/artifact/com.hadoop.gplcompression/hadoop-lzo/0.4.16
> Maybe you need to configure your `.m2/settings.xml` to have the repository,
> here's some doc from maven:
> https://maven.apache.org/guides/mini/guide-multiple-repositories.html
>
> Best wishes,
>
> Le mer. 3 avr. 2019 à 13:33, Andy Grove <An...@rms.com> a écrit :
>
> > I've been watching the RC process here and would like to help verify the
> > release, since I have co-workers wanting to upgrade to 1.11
> >
> > When I follow the instructions to verify the release I am running into an
> > issue resolving a dependency:
> >
> > Could not resolve dependencies for project
> > org.apache.parquet:parquet-thrift:jar:1.11.0: Could not find artifact
> > com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16
> >
> > I don't see anything in the README in the repo that explains how I should
> > configure maven to find this dependency. I don’t even see a direct
> > reference to this jar in maven, so would appreciate some guidance.
> >
> > Thanks,
> >
> > Andy,
> >
> >
> > On 3/21/19, 3:40 PM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
> >
> > CAUTION – UNVERIFIED EXTERNAL EMAIL
> >
> >
> > Hi Wes,
> >
> > Here is a list of steps (the first part is probably the same as for
> > parquet-cpp):
> >
> > 1. Download the files from
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=5WxAct62mkcX2OezQNaj15X9Q%2FQ3zJETYBNt2kKFyWk%3D&reserved=0
> > 2. Download the KEYS file from
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=yy4O7iXv8RZshf4qyarjyzPyIyNq4p3rtkjjNUWcIRY%3D&reserved=0
> > 3. Import the GPG keys by running `gpg --import KEYS`.
> > 4. Verify the signature by running `gpg --verify
> > apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature",
> the
> > warning about the key not being trusted can be ignored.
> > 5. Extract the source code: `tar xzf apache-parquet-1.11.0.tar.gz`.
> > 6. Change to the source directory: `cd apache-parquet-1.11.0`.
> > 7. Make sure that you have a thrift compiler and that you have the
> > correct
> > version by issuing `thrift -version`. It should print "Thrift version
> > 0.12.0". If you don't have it, you need to install it in a manner
> that
> > is
> > specific to your system (for example, `sudo apt install
> > thrift-compiler` or
> > `sudo yum install thrift-compiler`). If your package manager only
> > offers an
> > older version, you may still use that with the workaround shown
> below.
> > 8. Test the release by issuing `mvn clean verify`. If you have an
> older
> > thrift compiler, for example 0.9.3, you have to issue `mvn
> > -Dformat.thrift.version=0.9.3 -Dthrift.version=0.9.3 clean install`
> > instead. This may or may not work depending on how old your thrift
> > version
> > is (>= 0.9.0 should work). Running this command can take around 15
> > minutes
> > and should print "BUILD SUCCESS" at the end (followed by a few extra
> > lines).
> >
> > Br,
> >
> > Zoltan
> >
> > On Thu, Mar 21, 2019 at 7:58 PM Wes McKinney <we...@gmail.com>
> > wrote:
> >
> > > Are there any instructions written down about how to verify this
> > release?
> > >
> > > On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi
> > <zi...@cloudera.com.invalid>
> > > wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > > signature matches
> > > > git hash matches the git tag
> > > > source tarball matches the git tag
> > > > unit tests and integration tests pass
> > > >
> > > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> gabor@apache.org
> > >
> > > wrote:
> > > >
> > > > > Dear Parquet Users and Developers,
> > > > >
> > > > > I propose the following RC to be released as the official
> Apache
> > > > > Parquet 1.11.0 release:
> > > > >
> > > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > > *
> > > > >
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=MoO3FhiXqeXRt0jb0lvBqsNELuUUwKfjekX6DSvOr%2Fg%3D&reserved=0
> > > > >
> > > > > The release tarball, signature, and checksums are here:
> > > > > *
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=5WxAct62mkcX2OezQNaj15X9Q%2FQ3zJETYBNt2kKFyWk%3D&reserved=0
> > > > >
> > > > > You can find the KEYS file here:
> > > > > *
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fformatdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=e6W8VpBIil0xTYrzoJpNRR%2FCiKNClA87nxY3Y6UNmvM%3D&reserved=0
> > > <
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=yy4O7iXv8RZshf4qyarjyzPyIyNq4p3rtkjjNUWcIRY%3D&reserved=0
> > >
> > > > >
> > > > > Binary artifacts are staged in Nexus here:
> > > > > *
> > > > >
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=NAe4dRJv%2BNuFXnidnLD2Ljc7WuQxlryqbVZlwsNKLag%3D&reserved=0
> > > > >
> > > > > This release includes the following new features:
> > > > > - PARQUET-1201 - Column indexes
> > > > > - PARQUET-1253 - Support for new logical type representation
> > > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > > > - PARQUET-1388 - Nanosecond precision time and timestamp -
> > parquet-mr
> > > > >
> > > > > The release also includes bug fixes, including:
> > > > > - PARQUET-1472: Dictionary filter fails on
> FIXED_LEN_BYTE_ARRAY.
> > > > > - PARQUET-1510: Fix notEq for optional columns with null
> values.
> > > > > - PARQUET-1533: TestSnappy() throws OOM exception with
> > Parquet-1485
> > > change
> > > > > - PARQUET-1531: Page row count limit causes empty pages to be
> > written
> > > from
> > > > > MessageColumnIO
> > > > > - PARQUET-1544: Possible over-shading of modules
> > > > >
> > > > > The following change has been reverted so it is not part of any
> > public
> > > > > release:
> > > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > > > >
> > > > > Please download, verify, and test. The vote will be open for at
> > least
> > > 72
> > > > > hours.
> > > > >
> > > > > Thanks,
> > > > > Gabor
> > > > >
> > >
> >
> >
> >
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by XU Qinghui <qi...@gmail.com>.
Hello, Andy
I think this dependency is in maven repository:
https://mvnrepository.com/artifact/com.hadoop.gplcompression/hadoop-lzo/0.4.16
Maybe you need to configure your `.m2/settings.xml` to have the repository,
here's some doc from maven:
https://maven.apache.org/guides/mini/guide-multiple-repositories.html
Best wishes,
Le mer. 3 avr. 2019 à 13:33, Andy Grove <An...@rms.com> a écrit :
> I've been watching the RC process here and would like to help verify the
> release, since I have co-workers wanting to upgrade to 1.11
>
> When I follow the instructions to verify the release I am running into an
> issue resolving a dependency:
>
> Could not resolve dependencies for project
> org.apache.parquet:parquet-thrift:jar:1.11.0: Could not find artifact
> com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16
>
> I don't see anything in the README in the repo that explains how I should
> configure maven to find this dependency. I don’t even see a direct
> reference to this jar in maven, so would appreciate some guidance.
>
> Thanks,
>
> Andy,
>
>
> On 3/21/19, 3:40 PM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
>
> CAUTION – UNVERIFIED EXTERNAL EMAIL
>
>
> Hi Wes,
>
> Here is a list of steps (the first part is probably the same as for
> parquet-cpp):
>
> 1. Download the files from
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=5WxAct62mkcX2OezQNaj15X9Q%2FQ3zJETYBNt2kKFyWk%3D&reserved=0
> 2. Download the KEYS file from
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=yy4O7iXv8RZshf4qyarjyzPyIyNq4p3rtkjjNUWcIRY%3D&reserved=0
> 3. Import the GPG keys by running `gpg --import KEYS`.
> 4. Verify the signature by running `gpg --verify
> apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature", the
> warning about the key not being trusted can be ignored.
> 5. Extract the source code: `tar xzf apache-parquet-1.11.0.tar.gz`.
> 6. Change to the source directory: `cd apache-parquet-1.11.0`.
> 7. Make sure that you have a thrift compiler and that you have the
> correct
> version by issuing `thrift -version`. It should print "Thrift version
> 0.12.0". If you don't have it, you need to install it in a manner that
> is
> specific to your system (for example, `sudo apt install
> thrift-compiler` or
> `sudo yum install thrift-compiler`). If your package manager only
> offers an
> older version, you may still use that with the workaround shown below.
> 8. Test the release by issuing `mvn clean verify`. If you have an older
> thrift compiler, for example 0.9.3, you have to issue `mvn
> -Dformat.thrift.version=0.9.3 -Dthrift.version=0.9.3 clean install`
> instead. This may or may not work depending on how old your thrift
> version
> is (>= 0.9.0 should work). Running this command can take around 15
> minutes
> and should print "BUILD SUCCESS" at the end (followed by a few extra
> lines).
>
> Br,
>
> Zoltan
>
> On Thu, Mar 21, 2019 at 7:58 PM Wes McKinney <we...@gmail.com>
> wrote:
>
> > Are there any instructions written down about how to verify this
> release?
> >
> > On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi
> <zi...@cloudera.com.invalid>
> > wrote:
> > >
> > > +1 (binding)
> > >
> > > signature matches
> > > git hash matches the git tag
> > > source tarball matches the git tag
> > > unit tests and integration tests pass
> > >
> > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <gabor@apache.org
> >
> > wrote:
> > >
> > > > Dear Parquet Users and Developers,
> > > >
> > > > I propose the following RC to be released as the official Apache
> > > > Parquet 1.11.0 release:
> > > >
> > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > *
> > > >
> > > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=MoO3FhiXqeXRt0jb0lvBqsNELuUUwKfjekX6DSvOr%2Fg%3D&reserved=0
> > > >
> > > > The release tarball, signature, and checksums are here:
> > > > *
> > > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=5WxAct62mkcX2OezQNaj15X9Q%2FQ3zJETYBNt2kKFyWk%3D&reserved=0
> > > >
> > > > You can find the KEYS file here:
> > > > *
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fformatdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=e6W8VpBIil0xTYrzoJpNRR%2FCiKNClA87nxY3Y6UNmvM%3D&reserved=0
> > <
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=yy4O7iXv8RZshf4qyarjyzPyIyNq4p3rtkjjNUWcIRY%3D&reserved=0
> >
> > > >
> > > > Binary artifacts are staged in Nexus here:
> > > > *
> > > >
> > > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=NAe4dRJv%2BNuFXnidnLD2Ljc7WuQxlryqbVZlwsNKLag%3D&reserved=0
> > > >
> > > > This release includes the following new features:
> > > > - PARQUET-1201 - Column indexes
> > > > - PARQUET-1253 - Support for new logical type representation
> > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > > - PARQUET-1388 - Nanosecond precision time and timestamp -
> parquet-mr
> > > >
> > > > The release also includes bug fixes, including:
> > > > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> > > > - PARQUET-1510: Fix notEq for optional columns with null values.
> > > > - PARQUET-1533: TestSnappy() throws OOM exception with
> Parquet-1485
> > change
> > > > - PARQUET-1531: Page row count limit causes empty pages to be
> written
> > from
> > > > MessageColumnIO
> > > > - PARQUET-1544: Possible over-shading of modules
> > > >
> > > > The following change has been reverted so it is not part of any
> public
> > > > release:
> > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > > >
> > > > Please download, verify, and test. The vote will be open for at
> least
> > 72
> > > > hours.
> > > >
> > > > Thanks,
> > > > Gabor
> > > >
> >
>
>
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Andy Grove <An...@rms.com>.
I've been watching the RC process here and would like to help verify the release, since I have co-workers wanting to upgrade to 1.11
When I follow the instructions to verify the release I am running into an issue resolving a dependency:
Could not resolve dependencies for project org.apache.parquet:parquet-thrift:jar:1.11.0: Could not find artifact com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16
I don't see anything in the README in the repo that explains how I should configure maven to find this dependency. I don’t even see a direct reference to this jar in maven, so would appreciate some guidance.
Thanks,
Andy,
On 3/21/19, 3:40 PM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
CAUTION – UNVERIFIED EXTERNAL EMAIL
Hi Wes,
Here is a list of steps (the first part is probably the same as for
parquet-cpp):
1. Download the files from
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=5WxAct62mkcX2OezQNaj15X9Q%2FQ3zJETYBNt2kKFyWk%3D&reserved=0
2. Download the KEYS file from
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=yy4O7iXv8RZshf4qyarjyzPyIyNq4p3rtkjjNUWcIRY%3D&reserved=0
3. Import the GPG keys by running `gpg --import KEYS`.
4. Verify the signature by running `gpg --verify
apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature", the
warning about the key not being trusted can be ignored.
5. Extract the source code: `tar xzf apache-parquet-1.11.0.tar.gz`.
6. Change to the source directory: `cd apache-parquet-1.11.0`.
7. Make sure that you have a thrift compiler and that you have the correct
version by issuing `thrift -version`. It should print "Thrift version
0.12.0". If you don't have it, you need to install it in a manner that is
specific to your system (for example, `sudo apt install thrift-compiler` or
`sudo yum install thrift-compiler`). If your package manager only offers an
older version, you may still use that with the workaround shown below.
8. Test the release by issuing `mvn clean verify`. If you have an older
thrift compiler, for example 0.9.3, you have to issue `mvn
-Dformat.thrift.version=0.9.3 -Dthrift.version=0.9.3 clean install`
instead. This may or may not work depending on how old your thrift version
is (>= 0.9.0 should work). Running this command can take around 15 minutes
and should print "BUILD SUCCESS" at the end (followed by a few extra lines).
Br,
Zoltan
On Thu, Mar 21, 2019 at 7:58 PM Wes McKinney <we...@gmail.com> wrote:
> Are there any instructions written down about how to verify this release?
>
> On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi <zi...@cloudera.com.invalid>
> wrote:
> >
> > +1 (binding)
> >
> > signature matches
> > git hash matches the git tag
> > source tarball matches the git tag
> > unit tests and integration tests pass
> >
> > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <ga...@apache.org>
> wrote:
> >
> > > Dear Parquet Users and Developers,
> > >
> > > I propose the following RC to be released as the official Apache
> > > Parquet 1.11.0 release:
> > >
> > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > * This corresponds to the tag: apache-parquet-1.11.0
> > > *
> > >
> > >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=MoO3FhiXqeXRt0jb0lvBqsNELuUUwKfjekX6DSvOr%2Fg%3D&reserved=0
> > >
> > > The release tarball, signature, and checksums are here:
> > > *
> > >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=5WxAct62mkcX2OezQNaj15X9Q%2FQ3zJETYBNt2kKFyWk%3D&reserved=0
> > >
> > > You can find the KEYS file here:
> > > * https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fformatdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=e6W8VpBIil0xTYrzoJpNRR%2FCiKNClA87nxY3Y6UNmvM%3D&reserved=0
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=yy4O7iXv8RZshf4qyarjyzPyIyNq4p3rtkjjNUWcIRY%3D&reserved=0>
> > >
> > > Binary artifacts are staged in Nexus here:
> > > *
> > >
> > >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=NAe4dRJv%2BNuFXnidnLD2Ljc7WuQxlryqbVZlwsNKLag%3D&reserved=0
> > >
> > > This release includes the following new features:
> > > - PARQUET-1201 - Column indexes
> > > - PARQUET-1253 - Support for new logical type representation
> > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
> > >
> > > The release also includes bug fixes, including:
> > > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> > > - PARQUET-1510: Fix notEq for optional columns with null values.
> > > - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485
> change
> > > - PARQUET-1531: Page row count limit causes empty pages to be written
> from
> > > MessageColumnIO
> > > - PARQUET-1544: Possible over-shading of modules
> > >
> > > The following change has been reverted so it is not part of any public
> > > release:
> > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > >
> > > Please download, verify, and test. The vote will be open for at least
> 72
> > > hours.
> > >
> > > Thanks,
> > > Gabor
> > >
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Zoltan Ivanfi <zi...@cloudera.com.INVALID>.
PS: Oh, and Java 11 is not supported, only Java 8. You also need to have
mvn installed.
Zoltan
On Thu, Mar 21, 2019 at 10:40 PM Zoltan Ivanfi <zi...@cloudera.com> wrote:
> Hi Wes,
>
> Here is a list of steps (the first part is probably the same as for
> parquet-cpp):
>
> 1. Download the files from
> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
> 2. Download the KEYS file from
> https://dist.apache.org/repos/dist/dev/parquet/KEYS
> 3. Import the GPG keys by running `gpg --import KEYS`.
> 4. Verify the signature by running `gpg --verify
> apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature", the
> warning about the key not being trusted can be ignored.
> 5. Extract the source code: `tar xzf apache-parquet-1.11.0.tar.gz`.
> 6. Change to the source directory: `cd apache-parquet-1.11.0`.
> 7. Make sure that you have a thrift compiler and that you have the correct
> version by issuing `thrift -version`. It should print "Thrift version
> 0.12.0". If you don't have it, you need to install it in a manner that is
> specific to your system (for example, `sudo apt install thrift-compiler` or
> `sudo yum install thrift-compiler`). If your package manager only offers an
> older version, you may still use that with the workaround shown below.
> 8. Test the release by issuing `mvn clean verify`. If you have an older
> thrift compiler, for example 0.9.3, you have to issue `mvn
> -Dformat.thrift.version=0.9.3 -Dthrift.version=0.9.3 clean install`
> instead. This may or may not work depending on how old your thrift version
> is (>= 0.9.0 should work). Running this command can take around 15 minutes
> and should print "BUILD SUCCESS" at the end (followed by a few extra lines).
>
> Br,
>
> Zoltan
>
> On Thu, Mar 21, 2019 at 7:58 PM Wes McKinney <we...@gmail.com> wrote:
>
>> Are there any instructions written down about how to verify this release?
>>
>> On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi <zi...@cloudera.com.invalid>
>> wrote:
>> >
>> > +1 (binding)
>> >
>> > signature matches
>> > git hash matches the git tag
>> > source tarball matches the git tag
>> > unit tests and integration tests pass
>> >
>> > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <ga...@apache.org>
>> wrote:
>> >
>> > > Dear Parquet Users and Developers,
>> > >
>> > > I propose the following RC to be released as the official Apache
>> > > Parquet 1.11.0 release:
>> > >
>> > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
>> > > * This corresponds to the tag: apache-parquet-1.11.0
>> > > *
>> > >
>> > >
>> https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
>> > >
>> > > The release tarball, signature, and checksums are here:
>> > > *
>> > >
>> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
>> > >
>> > > You can find the KEYS file here:
>>
> > > * https://dist.apache.org/repos/formatdist/dev/parquet/KEYS
>> <https://dist.apache.org/repos/dist/dev/parquet/KEYS>
>
>
>> > >
>> > > Binary artifacts are staged in Nexus here:
>> > > *
>> > >
>> > >
>> https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
>> > >
>> > > This release includes the following new features:
>> > > - PARQUET-1201 - Column indexes
>> > > - PARQUET-1253 - Support for new logical type representation
>> > > - PARQUET-1381 - Add merge blocks command to parquet-tools
>> > > - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
>> > >
>> > > The release also includes bug fixes, including:
>> > > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
>> > > - PARQUET-1510: Fix notEq for optional columns with null values.
>> > > - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485
>> change
>> > > - PARQUET-1531: Page row count limit causes empty pages to be written
>> from
>> > > MessageColumnIO
>> > > - PARQUET-1544: Possible over-shading of modules
>> > >
>> > > The following change has been reverted so it is not part of any public
>> > > release:
>> > > - PARQUET-1381: Add merge blocks command to parquet-tools
>> > >
>> > > Please download, verify, and test. The vote will be open for at least
>> 72
>> > > hours.
>> > >
>> > > Thanks,
>> > > Gabor
>> > >
>>
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Zoltan Ivanfi <zi...@cloudera.com.INVALID>.
Hi Wes,
Here is a list of steps (the first part is probably the same as for
parquet-cpp):
1. Download the files from
https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
2. Download the KEYS file from
https://dist.apache.org/repos/dist/dev/parquet/KEYS
3. Import the GPG keys by running `gpg --import KEYS`.
4. Verify the signature by running `gpg --verify
apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature", the
warning about the key not being trusted can be ignored.
5. Extract the source code: `tar xzf apache-parquet-1.11.0.tar.gz`.
6. Change to the source directory: `cd apache-parquet-1.11.0`.
7. Make sure that you have a thrift compiler and that you have the correct
version by issuing `thrift -version`. It should print "Thrift version
0.12.0". If you don't have it, you need to install it in a manner that is
specific to your system (for example, `sudo apt install thrift-compiler` or
`sudo yum install thrift-compiler`). If your package manager only offers an
older version, you may still use that with the workaround shown below.
8. Test the release by issuing `mvn clean verify`. If you have an older
thrift compiler, for example 0.9.3, you have to issue `mvn
-Dformat.thrift.version=0.9.3 -Dthrift.version=0.9.3 clean install`
instead. This may or may not work depending on how old your thrift version
is (>= 0.9.0 should work). Running this command can take around 15 minutes
and should print "BUILD SUCCESS" at the end (followed by a few extra lines).
Br,
Zoltan
On Thu, Mar 21, 2019 at 7:58 PM Wes McKinney <we...@gmail.com> wrote:
> Are there any instructions written down about how to verify this release?
>
> On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi <zi...@cloudera.com.invalid>
> wrote:
> >
> > +1 (binding)
> >
> > signature matches
> > git hash matches the git tag
> > source tarball matches the git tag
> > unit tests and integration tests pass
> >
> > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <ga...@apache.org>
> wrote:
> >
> > > Dear Parquet Users and Developers,
> > >
> > > I propose the following RC to be released as the official Apache
> > > Parquet 1.11.0 release:
> > >
> > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > * This corresponds to the tag: apache-parquet-1.11.0
> > > *
> > >
> > >
> https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
> > >
> > > The release tarball, signature, and checksums are here:
> > > *
> > >
> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
> > >
> > > You can find the KEYS file here:
> > > * https://dist.apache.org/repos/formatdist/dev/parquet/KEYS
> <https://dist.apache.org/repos/dist/dev/parquet/KEYS>
> > >
> > > Binary artifacts are staged in Nexus here:
> > > *
> > >
> > >
> https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
> > >
> > > This release includes the following new features:
> > > - PARQUET-1201 - Column indexes
> > > - PARQUET-1253 - Support for new logical type representation
> > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
> > >
> > > The release also includes bug fixes, including:
> > > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> > > - PARQUET-1510: Fix notEq for optional columns with null values.
> > > - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485
> change
> > > - PARQUET-1531: Page row count limit causes empty pages to be written
> from
> > > MessageColumnIO
> > > - PARQUET-1544: Possible over-shading of modules
> > >
> > > The following change has been reverted so it is not part of any public
> > > release:
> > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > >
> > > Please download, verify, and test. The vote will be open for at least
> 72
> > > hours.
> > >
> > > Thanks,
> > > Gabor
> > >
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Wes McKinney <we...@gmail.com>.
Are there any instructions written down about how to verify this release?
On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi <zi...@cloudera.com.invalid> wrote:
>
> +1 (binding)
>
> signature matches
> git hash matches the git tag
> source tarball matches the git tag
> unit tests and integration tests pass
>
> On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <ga...@apache.org> wrote:
>
> > Dear Parquet Users and Developers,
> >
> > I propose the following RC to be released as the official Apache
> > Parquet 1.11.0 release:
> >
> > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > * This corresponds to the tag: apache-parquet-1.11.0
> > *
> >
> > https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
> >
> > The release tarball, signature, and checksums are here:
> > *
> > https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/dev/parquet/KEYS
> >
> > Binary artifacts are staged in Nexus here:
> > *
> >
> > https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
> >
> > This release includes the following new features:
> > - PARQUET-1201 - Column indexes
> > - PARQUET-1253 - Support for new logical type representation
> > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
> >
> > The release also includes bug fixes, including:
> > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> > - PARQUET-1510: Fix notEq for optional columns with null values.
> > - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485 change
> > - PARQUET-1531: Page row count limit causes empty pages to be written from
> > MessageColumnIO
> > - PARQUET-1544: Possible over-shading of modules
> >
> > The following change has been reverted so it is not part of any public
> > release:
> > - PARQUET-1381: Add merge blocks command to parquet-tools
> >
> > Please download, verify, and test. The vote will be open for at least 72
> > hours.
> >
> > Thanks,
> > Gabor
> >
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Zoltan Ivanfi <zi...@cloudera.com.INVALID>.
+1 (binding)
signature matches
git hash matches the git tag
source tarball matches the git tag
unit tests and integration tests pass
On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <ga...@apache.org> wrote:
> Dear Parquet Users and Developers,
>
> I propose the following RC to be released as the official Apache
> Parquet 1.11.0 release:
>
> The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> * This corresponds to the tag: apache-parquet-1.11.0
> *
>
> https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
>
> The release tarball, signature, and checksums are here:
> *
> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/parquet/KEYS
>
> Binary artifacts are staged in Nexus here:
> *
>
> https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
>
> This release includes the following new features:
> - PARQUET-1201 - Column indexes
> - PARQUET-1253 - Support for new logical type representation
> - PARQUET-1381 - Add merge blocks command to parquet-tools
> - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
>
> The release also includes bug fixes, including:
> - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> - PARQUET-1510: Fix notEq for optional columns with null values.
> - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485 change
> - PARQUET-1531: Page row count limit causes empty pages to be written from
> MessageColumnIO
> - PARQUET-1544: Possible over-shading of modules
>
> The following change has been reverted so it is not part of any public
> release:
> - PARQUET-1381: Add merge blocks command to parquet-tools
>
> Please download, verify, and test. The vote will be open for at least 72
> hours.
>
> Thanks,
> Gabor
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Ismaël Mejía <ie...@gmail.com>.
Thanks Fokko,
It seems then that getting Avro 1.9.1 is a better idea, but in that
case the question is still the same, can this be backported on 1.11.x,
eventually for 1.11.1 ? I ask this because after looking at Parquet's
release pace I am worried that we will have to wait to long to get it
in.
On Mon, Jun 3, 2019 at 3:26 PM Driesprong, Fokko <fo...@driesprong.frl> wrote:
>
> Ismaël,
>
> From what I understand from Jacob Tolar, I believe we have a blocking issue
> in Apache Avro: https://issues.apache.org/jira/browse/AVRO-2400 We would
> need to release 1.9.1 first to get this sorted out.
>
> Regarding Parquet 1.11.0, I would expect a branch which has been cut from
> master. However, I don't see this branch on Github.
>
> Cheers, Fokko
>
> Op ma 3 jun. 2019 om 15:19 schreef Ismaël Mejía <ie...@gmail.com>:
>
> > Any chance the upgrade to Avro 1.9.0 may be cherry picked (otherwise
> > it will take longer to get this downstreamed into Spark).
> >
> > I can work on the cherry pick if needed, just need to understand where
> > is the 1.11.x branch if we have one (or how is the procedure for this
> > in an ongoing release).
> > https://github.com/apache/parquet-mr/pull/638
> >
> >
> >
> > On Fri, May 31, 2019 at 6:25 PM Ryan Blue <rb...@netflix.com.invalid>
> > wrote:
> > >
> > > I think we can add that one.
> > >
> > > On Fri, May 31, 2019 at 9:18 AM Michael Heuer <he...@gmail.com> wrote:
> > >
> > > > Might
> > > >
> > > > https://github.com/apache/parquet-mr/pull/560
> > > >
> > > > be included in the next 1.11.0 release candidate?
> > > >
> > > > michael
> > > >
> > > >
> > > > On May 31, 2019, at 11:09 AM, Ryan Blue <rb...@netflix.com.INVALID>
> > wrote:
> > > >
> > > > I'm hoping to find some time to get a release candidate out next week
> > or
> > > > the week after.
> > > >
> > > > On Fri, May 31, 2019 at 4:32 AM Ismaël Mejía <ie...@apache.org>
> > wrote:
> > > >
> > > > Hello,
> > > >
> > > > What is the status of this, any chance we can have 1.11.0 out soon?
> > > >
> > > > Thanks,
> > > > Ismaël
> > > >
> > > >
> > > > On 2019/03/19 14:00:30, Gabor Szadovszky <ga...@apache.org> wrote:
> > > >
> > > > Dear Parquet Users and Developers,
> > > >
> > > > I propose the following RC to be released as the official Apache
> > > > Parquet 1.11.0 release:
> > > >
> > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > *
> > > >
> > > >
> > > >
> > https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
> > > >
> > > >
> > > > The release tarball, signature, and checksums are here:
> > > > *
> > > >
> > > >
> > https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
> > > >
> > > >
> > > > You can find the KEYS file here:
> > > > * https://dist.apache.org/repos/dist/dev/parquet/KEYS
> > > >
> > > > Binary artifacts are staged in Nexus here:
> > > > *
> > > >
> > > >
> > > >
> > https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
> > > >
> > > >
> > > > This release includes the following new features:
> > > > - PARQUET-1201 - Column indexes
> > > > - PARQUET-1253 - Support for new logical type representation
> > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > > - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
> > > >
> > > > The release also includes bug fixes, including:
> > > > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> > > > - PARQUET-1510: Fix notEq for optional columns with null values.
> > > > - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485
> > > >
> > > > change
> > > >
> > > > - PARQUET-1531: Page row count limit causes empty pages to be written
> > > >
> > > > from
> > > >
> > > > MessageColumnIO
> > > > - PARQUET-1544: Possible over-shading of modules
> > > >
> > > > The following change has been reverted so it is not part of any public
> > > > release:
> > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > > >
> > > > Please download, verify, and test. The vote will be open for at least
> > 72
> > > > hours.
> > > >
> > > > Thanks,
> > > > Gabor
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Ryan Blue
> > > > Software Engineer
> > > > Netflix
> > > >
> > > >
> > > >
> > >
> > > --
> > > Ryan Blue
> > > Software Engineer
> > > Netflix
> >
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Ismaël,
From what I understand from Jacob Tolar, I believe we have a blocking issue
in Apache Avro: https://issues.apache.org/jira/browse/AVRO-2400 We would
need to release 1.9.1 first to get this sorted out.
Regarding Parquet 1.11.0, I would expect a branch which has been cut from
master. However, I don't see this branch on Github.
Cheers, Fokko
Op ma 3 jun. 2019 om 15:19 schreef Ismaël Mejía <ie...@gmail.com>:
> Any chance the upgrade to Avro 1.9.0 may be cherry picked (otherwise
> it will take longer to get this downstreamed into Spark).
>
> I can work on the cherry pick if needed, just need to understand where
> is the 1.11.x branch if we have one (or how is the procedure for this
> in an ongoing release).
> https://github.com/apache/parquet-mr/pull/638
>
>
>
> On Fri, May 31, 2019 at 6:25 PM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
> >
> > I think we can add that one.
> >
> > On Fri, May 31, 2019 at 9:18 AM Michael Heuer <he...@gmail.com> wrote:
> >
> > > Might
> > >
> > > https://github.com/apache/parquet-mr/pull/560
> > >
> > > be included in the next 1.11.0 release candidate?
> > >
> > > michael
> > >
> > >
> > > On May 31, 2019, at 11:09 AM, Ryan Blue <rb...@netflix.com.INVALID>
> wrote:
> > >
> > > I'm hoping to find some time to get a release candidate out next week
> or
> > > the week after.
> > >
> > > On Fri, May 31, 2019 at 4:32 AM Ismaël Mejía <ie...@apache.org>
> wrote:
> > >
> > > Hello,
> > >
> > > What is the status of this, any chance we can have 1.11.0 out soon?
> > >
> > > Thanks,
> > > Ismaël
> > >
> > >
> > > On 2019/03/19 14:00:30, Gabor Szadovszky <ga...@apache.org> wrote:
> > >
> > > Dear Parquet Users and Developers,
> > >
> > > I propose the following RC to be released as the official Apache
> > > Parquet 1.11.0 release:
> > >
> > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > * This corresponds to the tag: apache-parquet-1.11.0
> > > *
> > >
> > >
> > >
> https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
> > >
> > >
> > > The release tarball, signature, and checksums are here:
> > > *
> > >
> > >
> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
> > >
> > >
> > > You can find the KEYS file here:
> > > * https://dist.apache.org/repos/dist/dev/parquet/KEYS
> > >
> > > Binary artifacts are staged in Nexus here:
> > > *
> > >
> > >
> > >
> https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
> > >
> > >
> > > This release includes the following new features:
> > > - PARQUET-1201 - Column indexes
> > > - PARQUET-1253 - Support for new logical type representation
> > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
> > >
> > > The release also includes bug fixes, including:
> > > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> > > - PARQUET-1510: Fix notEq for optional columns with null values.
> > > - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485
> > >
> > > change
> > >
> > > - PARQUET-1531: Page row count limit causes empty pages to be written
> > >
> > > from
> > >
> > > MessageColumnIO
> > > - PARQUET-1544: Possible over-shading of modules
> > >
> > > The following change has been reverted so it is not part of any public
> > > release:
> > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > >
> > > Please download, verify, and test. The vote will be open for at least
> 72
> > > hours.
> > >
> > > Thanks,
> > > Gabor
> > >
> > >
> > >
> > >
> > > --
> > > Ryan Blue
> > > Software Engineer
> > > Netflix
> > >
> > >
> > >
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Ismaël Mejía <ie...@gmail.com>.
Any chance the upgrade to Avro 1.9.0 may be cherry picked (otherwise
it will take longer to get this downstreamed into Spark).
I can work on the cherry pick if needed, just need to understand where
is the 1.11.x branch if we have one (or how is the procedure for this
in an ongoing release).
https://github.com/apache/parquet-mr/pull/638
On Fri, May 31, 2019 at 6:25 PM Ryan Blue <rb...@netflix.com.invalid> wrote:
>
> I think we can add that one.
>
> On Fri, May 31, 2019 at 9:18 AM Michael Heuer <he...@gmail.com> wrote:
>
> > Might
> >
> > https://github.com/apache/parquet-mr/pull/560
> >
> > be included in the next 1.11.0 release candidate?
> >
> > michael
> >
> >
> > On May 31, 2019, at 11:09 AM, Ryan Blue <rb...@netflix.com.INVALID> wrote:
> >
> > I'm hoping to find some time to get a release candidate out next week or
> > the week after.
> >
> > On Fri, May 31, 2019 at 4:32 AM Ismaël Mejía <ie...@apache.org> wrote:
> >
> > Hello,
> >
> > What is the status of this, any chance we can have 1.11.0 out soon?
> >
> > Thanks,
> > Ismaël
> >
> >
> > On 2019/03/19 14:00:30, Gabor Szadovszky <ga...@apache.org> wrote:
> >
> > Dear Parquet Users and Developers,
> >
> > I propose the following RC to be released as the official Apache
> > Parquet 1.11.0 release:
> >
> > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > * This corresponds to the tag: apache-parquet-1.11.0
> > *
> >
> >
> > https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
> >
> >
> > The release tarball, signature, and checksums are here:
> > *
> >
> > https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
> >
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/dev/parquet/KEYS
> >
> > Binary artifacts are staged in Nexus here:
> > *
> >
> >
> > https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
> >
> >
> > This release includes the following new features:
> > - PARQUET-1201 - Column indexes
> > - PARQUET-1253 - Support for new logical type representation
> > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
> >
> > The release also includes bug fixes, including:
> > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> > - PARQUET-1510: Fix notEq for optional columns with null values.
> > - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485
> >
> > change
> >
> > - PARQUET-1531: Page row count limit causes empty pages to be written
> >
> > from
> >
> > MessageColumnIO
> > - PARQUET-1544: Possible over-shading of modules
> >
> > The following change has been reverted so it is not part of any public
> > release:
> > - PARQUET-1381: Add merge blocks command to parquet-tools
> >
> > Please download, verify, and test. The vote will be open for at least 72
> > hours.
> >
> > Thanks,
> > Gabor
> >
> >
> >
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
> >
> >
> >
>
> --
> Ryan Blue
> Software Engineer
> Netflix
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
I think we can add that one.
On Fri, May 31, 2019 at 9:18 AM Michael Heuer <he...@gmail.com> wrote:
> Might
>
> https://github.com/apache/parquet-mr/pull/560
>
> be included in the next 1.11.0 release candidate?
>
> michael
>
>
> On May 31, 2019, at 11:09 AM, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>
> I'm hoping to find some time to get a release candidate out next week or
> the week after.
>
> On Fri, May 31, 2019 at 4:32 AM Ismaël Mejía <ie...@apache.org> wrote:
>
> Hello,
>
> What is the status of this, any chance we can have 1.11.0 out soon?
>
> Thanks,
> Ismaël
>
>
> On 2019/03/19 14:00:30, Gabor Szadovszky <ga...@apache.org> wrote:
>
> Dear Parquet Users and Developers,
>
> I propose the following RC to be released as the official Apache
> Parquet 1.11.0 release:
>
> The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> * This corresponds to the tag: apache-parquet-1.11.0
> *
>
>
> https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
>
>
> The release tarball, signature, and checksums are here:
> *
>
> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
>
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/parquet/KEYS
>
> Binary artifacts are staged in Nexus here:
> *
>
>
> https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
>
>
> This release includes the following new features:
> - PARQUET-1201 - Column indexes
> - PARQUET-1253 - Support for new logical type representation
> - PARQUET-1381 - Add merge blocks command to parquet-tools
> - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
>
> The release also includes bug fixes, including:
> - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> - PARQUET-1510: Fix notEq for optional columns with null values.
> - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485
>
> change
>
> - PARQUET-1531: Page row count limit causes empty pages to be written
>
> from
>
> MessageColumnIO
> - PARQUET-1544: Possible over-shading of modules
>
> The following change has been reverted so it is not part of any public
> release:
> - PARQUET-1381: Add merge blocks command to parquet-tools
>
> Please download, verify, and test. The vote will be open for at least 72
> hours.
>
> Thanks,
> Gabor
>
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
>
>
--
Ryan Blue
Software Engineer
Netflix
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Michael Heuer <he...@gmail.com>.
Might
https://github.com/apache/parquet-mr/pull/560 <https://github.com/apache/parquet-mr/pull/560>
be included in the next 1.11.0 release candidate?
michael
> On May 31, 2019, at 11:09 AM, Ryan Blue <rb...@netflix.com.INVALID> wrote:
>
> I'm hoping to find some time to get a release candidate out next week or
> the week after.
>
> On Fri, May 31, 2019 at 4:32 AM Ismaël Mejía <ie...@apache.org> wrote:
>
>> Hello,
>>
>> What is the status of this, any chance we can have 1.11.0 out soon?
>>
>> Thanks,
>> Ismaël
>>
>>
>> On 2019/03/19 14:00:30, Gabor Szadovszky <ga...@apache.org> wrote:
>>> Dear Parquet Users and Developers,
>>>
>>> I propose the following RC to be released as the official Apache
>>> Parquet 1.11.0 release:
>>>
>>> The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
>>> * This corresponds to the tag: apache-parquet-1.11.0
>>> *
>>>
>> https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
>>>
>>> The release tarball, signature, and checksums are here:
>>> *
>> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
>>>
>>> You can find the KEYS file here:
>>> * https://dist.apache.org/repos/dist/dev/parquet/KEYS
>>>
>>> Binary artifacts are staged in Nexus here:
>>> *
>>>
>> https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
>>>
>>> This release includes the following new features:
>>> - PARQUET-1201 - Column indexes
>>> - PARQUET-1253 - Support for new logical type representation
>>> - PARQUET-1381 - Add merge blocks command to parquet-tools
>>> - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
>>>
>>> The release also includes bug fixes, including:
>>> - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
>>> - PARQUET-1510: Fix notEq for optional columns with null values.
>>> - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485
>> change
>>> - PARQUET-1531: Page row count limit causes empty pages to be written
>> from
>>> MessageColumnIO
>>> - PARQUET-1544: Possible over-shading of modules
>>>
>>> The following change has been reverted so it is not part of any public
>>> release:
>>> - PARQUET-1381: Add merge blocks command to parquet-tools
>>>
>>> Please download, verify, and test. The vote will be open for at least 72
>>> hours.
>>>
>>> Thanks,
>>> Gabor
>>>
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
I'm hoping to find some time to get a release candidate out next week or
the week after.
On Fri, May 31, 2019 at 4:32 AM Ismaël Mejía <ie...@apache.org> wrote:
> Hello,
>
> What is the status of this, any chance we can have 1.11.0 out soon?
>
> Thanks,
> Ismaël
>
>
> On 2019/03/19 14:00:30, Gabor Szadovszky <ga...@apache.org> wrote:
> > Dear Parquet Users and Developers,
> >
> > I propose the following RC to be released as the official Apache
> > Parquet 1.11.0 release:
> >
> > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > * This corresponds to the tag: apache-parquet-1.11.0
> > *
> >
> https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
> >
> > The release tarball, signature, and checksums are here:
> > *
> https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/dev/parquet/KEYS
> >
> > Binary artifacts are staged in Nexus here:
> > *
> >
> https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
> >
> > This release includes the following new features:
> > - PARQUET-1201 - Column indexes
> > - PARQUET-1253 - Support for new logical type representation
> > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
> >
> > The release also includes bug fixes, including:
> > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> > - PARQUET-1510: Fix notEq for optional columns with null values.
> > - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485
> change
> > - PARQUET-1531: Page row count limit causes empty pages to be written
> from
> > MessageColumnIO
> > - PARQUET-1544: Possible over-shading of modules
> >
> > The following change has been reverted so it is not part of any public
> > release:
> > - PARQUET-1381: Add merge blocks command to parquet-tools
> >
> > Please download, verify, and test. The vote will be open for at least 72
> > hours.
> >
> > Thanks,
> > Gabor
> >
>
--
Ryan Blue
Software Engineer
Netflix
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Isma��l Mej��a <ie...@apache.org>.
Hello,
What is the status of this, any chance we can have 1.11.0 out soon?
Thanks,
Ismaël
On 2019/03/19 14:00:30, Gabor Szadovszky <ga...@apache.org> wrote:
> Dear Parquet Users and Developers,
>
> I propose the following RC to be released as the official Apache
> Parquet 1.11.0 release:
>
> The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> * This corresponds to the tag: apache-parquet-1.11.0
> *
> https://github.com/apache/parquet-mr/tree/9756b0e2b35437a09716707a81e2ac0c187112ed
>
> The release tarball, signature, and checksums are here:
> * https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.11.0-rc6/
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/parquet/KEYS
>
> Binary artifacts are staged in Nexus here:
> *
> https://repository.apache.org/content/groups/staging/org/apache/parquet/parquet/1.11.0/
>
> This release includes the following new features:
> - PARQUET-1201 - Column indexes
> - PARQUET-1253 - Support for new logical type representation
> - PARQUET-1381 - Add merge blocks command to parquet-tools
> - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
>
> The release also includes bug fixes, including:
> - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> - PARQUET-1510: Fix notEq for optional columns with null values.
> - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485 change
> - PARQUET-1531: Page row count limit causes empty pages to be written from
> MessageColumnIO
> - PARQUET-1544: Possible over-shading of modules
>
> The following change has been reverted so it is not part of any public
> release:
> - PARQUET-1381: Add merge blocks command to parquet-tools
>
> Please download, verify, and test. The vote will be open for at least 72
> hours.
>
> Thanks,
> Gabor
>