You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Andy Grove <An...@rms.com> on 2019/04/03 11:27:11 UTC
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
I've been watching the RC process here and would like to help verify the release, since I have co-workers wanting to upgrade to 1.11
When I follow the instructions to verify the release I am running into an issue resolving a dependency:
Could not resolve dependencies for project org.apache.parquet:parquet-thrift:jar:1.11.0: Could not find artifact com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16
I don't see anything in the README in the repo that explains how I should configure maven to find this dependency. I don’t even see a direct reference to this jar in maven, so would appreciate some guidance.
Thanks,
Andy,
On 3/21/19, 3:40 PM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
CAUTION – UNVERIFIED EXTERNAL EMAIL
Hi Wes,
Here is a list of steps (the first part is probably the same as for
parquet-cpp):
1. Download the files from
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=5WxAct62mkcX2OezQNaj15X9Q%2FQ3zJETYBNt2kKFyWk%3D&reserved=0
2. Download the KEYS file from
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=yy4O7iXv8RZshf4qyarjyzPyIyNq4p3rtkjjNUWcIRY%3D&reserved=0
3. Import the GPG keys by running `gpg --import KEYS`.
4. Verify the signature by running `gpg --verify
apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature", the
warning about the key not being trusted can be ignored.
5. Extract the source code: `tar xzf apache-parquet-1.11.0.tar.gz`.
6. Change to the source directory: `cd apache-parquet-1.11.0`.
7. Make sure that you have a thrift compiler and that you have the correct
version by issuing `thrift -version`. It should print "Thrift version
0.12.0". If you don't have it, you need to install it in a manner that is
specific to your system (for example, `sudo apt install thrift-compiler` or
`sudo yum install thrift-compiler`). If your package manager only offers an
older version, you may still use that with the workaround shown below.
8. Test the release by issuing `mvn clean verify`. If you have an older
thrift compiler, for example 0.9.3, you have to issue `mvn
-Dformat.thrift.version=0.9.3 -Dthrift.version=0.9.3 clean install`
instead. This may or may not work depending on how old your thrift version
is (>= 0.9.0 should work). Running this command can take around 15 minutes
and should print "BUILD SUCCESS" at the end (followed by a few extra lines).
Br,
Zoltan
On Thu, Mar 21, 2019 at 7:58 PM Wes McKinney <we...@gmail.com> wrote:
> Are there any instructions written down about how to verify this release?
>
> On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi <zi...@cloudera.com.invalid>
> wrote:
> >
> > +1 (binding)
> >
> > signature matches
> > git hash matches the git tag
> > source tarball matches the git tag
> > unit tests and integration tests pass
> >
> > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <ga...@apache.org>
> wrote:
> >
> > > Dear Parquet Users and Developers,
> > >
> > > I propose the following RC to be released as the official Apache
> > > Parquet 1.11.0 release:
> > >
> > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > * This corresponds to the tag: apache-parquet-1.11.0
> > > *
> > >
> > >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=MoO3FhiXqeXRt0jb0lvBqsNELuUUwKfjekX6DSvOr%2Fg%3D&reserved=0
> > >
> > > The release tarball, signature, and checksums are here:
> > > *
> > >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=5WxAct62mkcX2OezQNaj15X9Q%2FQ3zJETYBNt2kKFyWk%3D&reserved=0
> > >
> > > You can find the KEYS file here:
> > > * https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fformatdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=e6W8VpBIil0xTYrzoJpNRR%2FCiKNClA87nxY3Y6UNmvM%3D&reserved=0
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=yy4O7iXv8RZshf4qyarjyzPyIyNq4p3rtkjjNUWcIRY%3D&reserved=0>
> > >
> > > Binary artifacts are staged in Nexus here:
> > > *
> > >
> > >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=NAe4dRJv%2BNuFXnidnLD2Ljc7WuQxlryqbVZlwsNKLag%3D&reserved=0
> > >
> > > This release includes the following new features:
> > > - PARQUET-1201 - Column indexes
> > > - PARQUET-1253 - Support for new logical type representation
> > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > - PARQUET-1388 - Nanosecond precision time and timestamp - parquet-mr
> > >
> > > The release also includes bug fixes, including:
> > > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> > > - PARQUET-1510: Fix notEq for optional columns with null values.
> > > - PARQUET-1533: TestSnappy() throws OOM exception with Parquet-1485
> change
> > > - PARQUET-1531: Page row count limit causes empty pages to be written
> from
> > > MessageColumnIO
> > > - PARQUET-1544: Possible over-shading of modules
> > >
> > > The following change has been reverted so it is not part of any public
> > > release:
> > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > >
> > > Please download, verify, and test. The vote will be open for at least
> 72
> > > hours.
> > >
> > > Thanks,
> > > Gabor
> > >
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Zoltan Ivanfi <zi...@cloudera.com.INVALID>.
Hi,
I have made some experiments as well and this gets more and more
interesting. Let's move this conversation to the JIRA though (thanks for
creating it by the way).
Br,
Zoltan
On Wed, Apr 3, 2019 at 5:17 PM Andy Grove <An...@rms.com> wrote:
> It works after configuring the twitter Maven repo. Here is the JIRA:
> https://issues.apache.org/jira/browse/PARQUET-1556
>
> On 4/3/19, 9:07 AM, "Andy Grove" <An...@rms.com> wrote:
>
> CAUTION – UNVERIFIED EXTERNAL EMAIL
>
>
> This artifact isn't available in maven central. It is in Twitter's
> maven repo.
>
> I am working on setting up my m2 settings to be able to read from
> there, but this is something that really needs to be documented.
>
> Once I figure it out, I will create a JIRA + PR to update the README.
>
> Thanks.
>
> On 4/3/19, 8:55 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
>
> CAUTION – UNVERIFIED EXTERNAL EMAIL
>
>
> Hi,
>
> That error message is quite surprising, Maven should be able to
> find that
> artifact without any manual configuration. It may have been some
> temporary
> network issue with accessing the repository. Could you try it
> again? If it
> keeps failing, Maven may have cached the unavailability of that
> artifact,
> which can be cleaned by issuing
>
> mvn dependency:purge-local-repository
> -DmanualInclude=com.hadoop.gplcompression:hadoop-lzo
>
> In case you still continue to get the same failure, you may be
> behind a
> stale artifact cache or some other kind of proxy or that does not
> allow you
> to access this artifact directly.
>
> Br,
>
> Zoltan
>
>
> On Wed, Apr 3, 2019 at 2:14 PM XU Qinghui <qi...@gmail.com>
> wrote:
>
> > Hello, Andy
> >
> > I think this dependency is in maven repository:
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmvnrepository.com%2Fartifact%2Fcom.hadoop.gplcompression%2Fhadoop-lzo%2F0.4.16&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=Bpzt4O1bvNXqy%2FWO7KKYh2sECoU6KV94fZXvIZ5M0Tk%3D&reserved=0
> > Maybe you need to configure your `.m2/settings.xml` to have the
> repository,
> > here's some doc from maven:
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaven.apache.org%2Fguides%2Fmini%2Fguide-multiple-repositories.html&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=DwAmqo7IPC7EYNZE03b76EKYdzruQ%2FI6NFpPmLEELOo%3D&reserved=0
> >
> > Best wishes,
> >
> > Le mer. 3 avr. 2019 à 13:33, Andy Grove <An...@rms.com> a
> écrit :
> >
> > > I've been watching the RC process here and would like to help
> verify the
> > > release, since I have co-workers wanting to upgrade to 1.11
> > >
> > > When I follow the instructions to verify the release I am
> running into an
> > > issue resolving a dependency:
> > >
> > > Could not resolve dependencies for project
> > > org.apache.parquet:parquet-thrift:jar:1.11.0: Could not find
> artifact
> > > com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16
> > >
> > > I don't see anything in the README in the repo that explains
> how I should
> > > configure maven to find this dependency. I don’t even see a
> direct
> > > reference to this jar in maven, so would appreciate some
> guidance.
> > >
> > > Thanks,
> > >
> > > Andy,
> > >
> > >
> > > On 3/21/19, 3:40 PM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID>
> wrote:
> > >
> > > CAUTION – UNVERIFIED EXTERNAL EMAIL
> > >
> > >
> > > Hi Wes,
> > >
> > > Here is a list of steps (the first part is probably the
> same as for
> > > parquet-cpp):
> > >
> > > 1. Download the files from
> > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=09cNlF5%2FzvJwIqAT9tsixh82aoJEna6h1voR7C3S%2BXc%3D&reserved=0
> > > 2. Download the KEYS file from
> > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=Dad6aN4TzDgVmoRDrAxDvTxEO6qFyVYGkHlVvK6ez%2Bo%3D&reserved=0
> > > 3. Import the GPG keys by running `gpg --import KEYS`.
> > > 4. Verify the signature by running `gpg --verify
> > > apache-parquet-1.11.0.tar.gz.asc`. It should say "Good
> signature",
> > the
> > > warning about the key not being trusted can be ignored.
> > > 5. Extract the source code: `tar xzf
> apache-parquet-1.11.0.tar.gz`.
> > > 6. Change to the source directory: `cd
> apache-parquet-1.11.0`.
> > > 7. Make sure that you have a thrift compiler and that you
> have the
> > > correct
> > > version by issuing `thrift -version`. It should print
> "Thrift version
> > > 0.12.0". If you don't have it, you need to install it in a
> manner
> > that
> > > is
> > > specific to your system (for example, `sudo apt install
> > > thrift-compiler` or
> > > `sudo yum install thrift-compiler`). If your package
> manager only
> > > offers an
> > > older version, you may still use that with the workaround
> shown
> > below.
> > > 8. Test the release by issuing `mvn clean verify`. If you
> have an
> > older
> > > thrift compiler, for example 0.9.3, you have to issue `mvn
> > > -Dformat.thrift.version=0.9.3 -Dthrift.version=0.9.3 clean
> install`
> > > instead. This may or may not work depending on how old
> your thrift
> > > version
> > > is (>= 0.9.0 should work). Running this command can take
> around 15
> > > minutes
> > > and should print "BUILD SUCCESS" at the end (followed by a
> few extra
> > > lines).
> > >
> > > Br,
> > >
> > > Zoltan
> > >
> > > On Thu, Mar 21, 2019 at 7:58 PM Wes McKinney <
> wesmckinn@gmail.com>
> > > wrote:
> > >
> > > > Are there any instructions written down about how to
> verify this
> > > release?
> > > >
> > > > On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi
> > > <zi...@cloudera.com.invalid>
> > > > wrote:
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > signature matches
> > > > > git hash matches the git tag
> > > > > source tarball matches the git tag
> > > > > unit tests and integration tests pass
> > > > >
> > > > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> > gabor@apache.org
> > > >
> > > > wrote:
> > > > >
> > > > > > Dear Parquet Users and Developers,
> > > > > >
> > > > > > I propose the following RC to be released as the
> official
> > Apache
> > > > > > Parquet 1.11.0 release:
> > > > > >
> > > > > > The commit id is
> 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > > > *
> > > > > >
> > > > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=ZKYiD%2F%2FDwP7zDadpxBvtSF4T%2FjXCdwIsAT8w%2FBQcxAQ%3D&reserved=0
> > > > > >
> > > > > > The release tarball, signature, and checksums are
> here:
> > > > > > *
> > > > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=v9PAI9BCY0HTGld7Zlb%2F8gJYbQH%2Fy1WmOw87ZzWymUU%3D&reserved=0
> > > > > >
> > > > > > You can find the KEYS file here:
> > > > > > *
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fformatdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=VqkSY5roEqYKo%2B09o6Wh2NZQWKkdbAKDvtWIdbLjccE%3D&reserved=0
> > > > <
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=2VCbJrdrUE1g3dN3VUqN%2FAq1Gt6CcBAGmGfx3CjlauI%3D&reserved=0
> > > >
> > > > > >
> > > > > > Binary artifacts are staged in Nexus here:
> > > > > > *
> > > > > >
> > > > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=Ff9wOSvCIG25l9N89STbYJfod7Oo0cb9pWqfOlaRD9s%3D&reserved=0
> > > > > >
> > > > > > This release includes the following new features:
> > > > > > - PARQUET-1201 - Column indexes
> > > > > > - PARQUET-1253 - Support for new logical type
> representation
> > > > > > - PARQUET-1381 - Add merge blocks command to
> parquet-tools
> > > > > > - PARQUET-1388 - Nanosecond precision time and
> timestamp -
> > > parquet-mr
> > > > > >
> > > > > > The release also includes bug fixes, including:
> > > > > > - PARQUET-1472: Dictionary filter fails on
> > FIXED_LEN_BYTE_ARRAY.
> > > > > > - PARQUET-1510: Fix notEq for optional columns with
> null
> > values.
> > > > > > - PARQUET-1533: TestSnappy() throws OOM exception
> with
> > > Parquet-1485
> > > > change
> > > > > > - PARQUET-1531: Page row count limit causes empty
> pages to be
> > > written
> > > > from
> > > > > > MessageColumnIO
> > > > > > - PARQUET-1544: Possible over-shading of modules
> > > > > >
> > > > > > The following change has been reverted so it is not
> part of any
> > > public
> > > > > > release:
> > > > > > - PARQUET-1381: Add merge blocks command to
> parquet-tools
> > > > > >
> > > > > > Please download, verify, and test. The vote will be
> open for at
> > > least
> > > > 72
> > > > > > hours.
> > > > > >
> > > > > > Thanks,
> > > > > > Gabor
> > > > > >
> > > >
> > >
> > >
> > >
> >
>
>
>
>
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Andy Grove <An...@rms.com>.
It works after configuring the twitter Maven repo. Here is the JIRA: https://issues.apache.org/jira/browse/PARQUET-1556
On 4/3/19, 9:07 AM, "Andy Grove" <An...@rms.com> wrote:
CAUTION – UNVERIFIED EXTERNAL EMAIL
This artifact isn't available in maven central. It is in Twitter's maven repo.
I am working on setting up my m2 settings to be able to read from there, but this is something that really needs to be documented.
Once I figure it out, I will create a JIRA + PR to update the README.
Thanks.
On 4/3/19, 8:55 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
CAUTION – UNVERIFIED EXTERNAL EMAIL
Hi,
That error message is quite surprising, Maven should be able to find that
artifact without any manual configuration. It may have been some temporary
network issue with accessing the repository. Could you try it again? If it
keeps failing, Maven may have cached the unavailability of that artifact,
which can be cleaned by issuing
mvn dependency:purge-local-repository
-DmanualInclude=com.hadoop.gplcompression:hadoop-lzo
In case you still continue to get the same failure, you may be behind a
stale artifact cache or some other kind of proxy or that does not allow you
to access this artifact directly.
Br,
Zoltan
On Wed, Apr 3, 2019 at 2:14 PM XU Qinghui <qi...@gmail.com> wrote:
> Hello, Andy
>
> I think this dependency is in maven repository:
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmvnrepository.com%2Fartifact%2Fcom.hadoop.gplcompression%2Fhadoop-lzo%2F0.4.16&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=Bpzt4O1bvNXqy%2FWO7KKYh2sECoU6KV94fZXvIZ5M0Tk%3D&reserved=0
> Maybe you need to configure your `.m2/settings.xml` to have the repository,
> here's some doc from maven:
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaven.apache.org%2Fguides%2Fmini%2Fguide-multiple-repositories.html&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=DwAmqo7IPC7EYNZE03b76EKYdzruQ%2FI6NFpPmLEELOo%3D&reserved=0
>
> Best wishes,
>
> Le mer. 3 avr. 2019 à 13:33, Andy Grove <An...@rms.com> a écrit :
>
> > I've been watching the RC process here and would like to help verify the
> > release, since I have co-workers wanting to upgrade to 1.11
> >
> > When I follow the instructions to verify the release I am running into an
> > issue resolving a dependency:
> >
> > Could not resolve dependencies for project
> > org.apache.parquet:parquet-thrift:jar:1.11.0: Could not find artifact
> > com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16
> >
> > I don't see anything in the README in the repo that explains how I should
> > configure maven to find this dependency. I don’t even see a direct
> > reference to this jar in maven, so would appreciate some guidance.
> >
> > Thanks,
> >
> > Andy,
> >
> >
> > On 3/21/19, 3:40 PM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
> >
> > CAUTION – UNVERIFIED EXTERNAL EMAIL
> >
> >
> > Hi Wes,
> >
> > Here is a list of steps (the first part is probably the same as for
> > parquet-cpp):
> >
> > 1. Download the files from
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=09cNlF5%2FzvJwIqAT9tsixh82aoJEna6h1voR7C3S%2BXc%3D&reserved=0
> > 2. Download the KEYS file from
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=Dad6aN4TzDgVmoRDrAxDvTxEO6qFyVYGkHlVvK6ez%2Bo%3D&reserved=0
> > 3. Import the GPG keys by running `gpg --import KEYS`.
> > 4. Verify the signature by running `gpg --verify
> > apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature",
> the
> > warning about the key not being trusted can be ignored.
> > 5. Extract the source code: `tar xzf apache-parquet-1.11.0.tar.gz`.
> > 6. Change to the source directory: `cd apache-parquet-1.11.0`.
> > 7. Make sure that you have a thrift compiler and that you have the
> > correct
> > version by issuing `thrift -version`. It should print "Thrift version
> > 0.12.0". If you don't have it, you need to install it in a manner
> that
> > is
> > specific to your system (for example, `sudo apt install
> > thrift-compiler` or
> > `sudo yum install thrift-compiler`). If your package manager only
> > offers an
> > older version, you may still use that with the workaround shown
> below.
> > 8. Test the release by issuing `mvn clean verify`. If you have an
> older
> > thrift compiler, for example 0.9.3, you have to issue `mvn
> > -Dformat.thrift.version=0.9.3 -Dthrift.version=0.9.3 clean install`
> > instead. This may or may not work depending on how old your thrift
> > version
> > is (>= 0.9.0 should work). Running this command can take around 15
> > minutes
> > and should print "BUILD SUCCESS" at the end (followed by a few extra
> > lines).
> >
> > Br,
> >
> > Zoltan
> >
> > On Thu, Mar 21, 2019 at 7:58 PM Wes McKinney <we...@gmail.com>
> > wrote:
> >
> > > Are there any instructions written down about how to verify this
> > release?
> > >
> > > On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi
> > <zi...@cloudera.com.invalid>
> > > wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > > signature matches
> > > > git hash matches the git tag
> > > > source tarball matches the git tag
> > > > unit tests and integration tests pass
> > > >
> > > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> gabor@apache.org
> > >
> > > wrote:
> > > >
> > > > > Dear Parquet Users and Developers,
> > > > >
> > > > > I propose the following RC to be released as the official
> Apache
> > > > > Parquet 1.11.0 release:
> > > > >
> > > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > > *
> > > > >
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585976838&sdata=ZKYiD%2F%2FDwP7zDadpxBvtSF4T%2FjXCdwIsAT8w%2FBQcxAQ%3D&reserved=0
> > > > >
> > > > > The release tarball, signature, and checksums are here:
> > > > > *
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=v9PAI9BCY0HTGld7Zlb%2F8gJYbQH%2Fy1WmOw87ZzWymUU%3D&reserved=0
> > > > >
> > > > > You can find the KEYS file here:
> > > > > *
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fformatdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=VqkSY5roEqYKo%2B09o6Wh2NZQWKkdbAKDvtWIdbLjccE%3D&reserved=0
> > > <
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=2VCbJrdrUE1g3dN3VUqN%2FAq1Gt6CcBAGmGfx3CjlauI%3D&reserved=0
> > >
> > > > >
> > > > > Binary artifacts are staged in Nexus here:
> > > > > *
> > > > >
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C0cd3c43dceca4c34f53108d6b8461b90%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899008585986852&sdata=Ff9wOSvCIG25l9N89STbYJfod7Oo0cb9pWqfOlaRD9s%3D&reserved=0
> > > > >
> > > > > This release includes the following new features:
> > > > > - PARQUET-1201 - Column indexes
> > > > > - PARQUET-1253 - Support for new logical type representation
> > > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > > > - PARQUET-1388 - Nanosecond precision time and timestamp -
> > parquet-mr
> > > > >
> > > > > The release also includes bug fixes, including:
> > > > > - PARQUET-1472: Dictionary filter fails on
> FIXED_LEN_BYTE_ARRAY.
> > > > > - PARQUET-1510: Fix notEq for optional columns with null
> values.
> > > > > - PARQUET-1533: TestSnappy() throws OOM exception with
> > Parquet-1485
> > > change
> > > > > - PARQUET-1531: Page row count limit causes empty pages to be
> > written
> > > from
> > > > > MessageColumnIO
> > > > > - PARQUET-1544: Possible over-shading of modules
> > > > >
> > > > > The following change has been reverted so it is not part of any
> > public
> > > > > release:
> > > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > > > >
> > > > > Please download, verify, and test. The vote will be open for at
> > least
> > > 72
> > > > > hours.
> > > > >
> > > > > Thanks,
> > > > > Gabor
> > > > >
> > >
> >
> >
> >
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Andy Grove <An...@rms.com>.
This artifact isn't available in maven central. It is in Twitter's maven repo.
I am working on setting up my m2 settings to be able to read from there, but this is something that really needs to be documented.
Once I figure it out, I will create a JIRA + PR to update the README.
Thanks.
On 4/3/19, 8:55 AM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
CAUTION – UNVERIFIED EXTERNAL EMAIL
Hi,
That error message is quite surprising, Maven should be able to find that
artifact without any manual configuration. It may have been some temporary
network issue with accessing the repository. Could you try it again? If it
keeps failing, Maven may have cached the unavailability of that artifact,
which can be cleaned by issuing
mvn dependency:purge-local-repository
-DmanualInclude=com.hadoop.gplcompression:hadoop-lzo
In case you still continue to get the same failure, you may be behind a
stale artifact cache or some other kind of proxy or that does not allow you
to access this artifact directly.
Br,
Zoltan
On Wed, Apr 3, 2019 at 2:14 PM XU Qinghui <qi...@gmail.com> wrote:
> Hello, Andy
>
> I think this dependency is in maven repository:
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmvnrepository.com%2Fartifact%2Fcom.hadoop.gplcompression%2Fhadoop-lzo%2F0.4.16&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001407997784&sdata=RpIKEzypbGLJFh0%2FgqDBKlkYoMTS6SJPQer95PszzZA%3D&reserved=0
> Maybe you need to configure your `.m2/settings.xml` to have the repository,
> here's some doc from maven:
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaven.apache.org%2Fguides%2Fmini%2Fguide-multiple-repositories.html&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001407997784&sdata=PEQr6hvv1dnNi5p6WAtU5o0xjvx3X9CqnTTV0ofVsjg%3D&reserved=0
>
> Best wishes,
>
> Le mer. 3 avr. 2019 à 13:33, Andy Grove <An...@rms.com> a écrit :
>
> > I've been watching the RC process here and would like to help verify the
> > release, since I have co-workers wanting to upgrade to 1.11
> >
> > When I follow the instructions to verify the release I am running into an
> > issue resolving a dependency:
> >
> > Could not resolve dependencies for project
> > org.apache.parquet:parquet-thrift:jar:1.11.0: Could not find artifact
> > com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16
> >
> > I don't see anything in the README in the repo that explains how I should
> > configure maven to find this dependency. I don’t even see a direct
> > reference to this jar in maven, so would appreciate some guidance.
> >
> > Thanks,
> >
> > Andy,
> >
> >
> > On 3/21/19, 3:40 PM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
> >
> > CAUTION – UNVERIFIED EXTERNAL EMAIL
> >
> >
> > Hi Wes,
> >
> > Here is a list of steps (the first part is probably the same as for
> > parquet-cpp):
> >
> > 1. Download the files from
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001407997784&sdata=7PYUYgwmDw%2BqJplXViaytfw2%2FQQOh00TN7FEHGKhJtw%3D&reserved=0
> > 2. Download the KEYS file from
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001407997784&sdata=drV7YZLYJOsaK1ta%2Bfu4gguxmvGP20YzFbJVVdXtF0M%3D&reserved=0
> > 3. Import the GPG keys by running `gpg --import KEYS`.
> > 4. Verify the signature by running `gpg --verify
> > apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature",
> the
> > warning about the key not being trusted can be ignored.
> > 5. Extract the source code: `tar xzf apache-parquet-1.11.0.tar.gz`.
> > 6. Change to the source directory: `cd apache-parquet-1.11.0`.
> > 7. Make sure that you have a thrift compiler and that you have the
> > correct
> > version by issuing `thrift -version`. It should print "Thrift version
> > 0.12.0". If you don't have it, you need to install it in a manner
> that
> > is
> > specific to your system (for example, `sudo apt install
> > thrift-compiler` or
> > `sudo yum install thrift-compiler`). If your package manager only
> > offers an
> > older version, you may still use that with the workaround shown
> below.
> > 8. Test the release by issuing `mvn clean verify`. If you have an
> older
> > thrift compiler, for example 0.9.3, you have to issue `mvn
> > -Dformat.thrift.version=0.9.3 -Dthrift.version=0.9.3 clean install`
> > instead. This may or may not work depending on how old your thrift
> > version
> > is (>= 0.9.0 should work). Running this command can take around 15
> > minutes
> > and should print "BUILD SUCCESS" at the end (followed by a few extra
> > lines).
> >
> > Br,
> >
> > Zoltan
> >
> > On Thu, Mar 21, 2019 at 7:58 PM Wes McKinney <we...@gmail.com>
> > wrote:
> >
> > > Are there any instructions written down about how to verify this
> > release?
> > >
> > > On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi
> > <zi...@cloudera.com.invalid>
> > > wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > > signature matches
> > > > git hash matches the git tag
> > > > source tarball matches the git tag
> > > > unit tests and integration tests pass
> > > >
> > > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> gabor@apache.org
> > >
> > > wrote:
> > > >
> > > > > Dear Parquet Users and Developers,
> > > > >
> > > > > I propose the following RC to be released as the official
> Apache
> > > > > Parquet 1.11.0 release:
> > > > >
> > > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > > *
> > > > >
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001408007793&sdata=oFkElFs3tclPAJwoxB1nBv%2FlNcLFxHIYPQVFtOdikGc%3D&reserved=0
> > > > >
> > > > > The release tarball, signature, and checksums are here:
> > > > > *
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001408007793&sdata=hbTVHQfazBMEsNKJSs2tk0nUlSTJ800rZgzwjC8tTbk%3D&reserved=0
> > > > >
> > > > > You can find the KEYS file here:
> > > > > *
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fformatdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001408007793&sdata=peJjszgj7vBJpxTn%2BglNiuAgRf2bL0%2FK0Md5gLl8A7s%3D&reserved=0
> > > <
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001408007793&sdata=GxLCnU0NjoyXZ3XJCRI0QPlJyUTyus814k%2FironJ7PA%3D&reserved=0
> > >
> > > > >
> > > > > Binary artifacts are staged in Nexus here:
> > > > > *
> > > > >
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C7a7c80c731264f8134d708d6b8446ff4%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636899001408007793&sdata=HJZSphBPPRw81jLJ%2FqBHfT4TlqAsGBhCNGWQIr8LNlE%3D&reserved=0
> > > > >
> > > > > This release includes the following new features:
> > > > > - PARQUET-1201 - Column indexes
> > > > > - PARQUET-1253 - Support for new logical type representation
> > > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > > > - PARQUET-1388 - Nanosecond precision time and timestamp -
> > parquet-mr
> > > > >
> > > > > The release also includes bug fixes, including:
> > > > > - PARQUET-1472: Dictionary filter fails on
> FIXED_LEN_BYTE_ARRAY.
> > > > > - PARQUET-1510: Fix notEq for optional columns with null
> values.
> > > > > - PARQUET-1533: TestSnappy() throws OOM exception with
> > Parquet-1485
> > > change
> > > > > - PARQUET-1531: Page row count limit causes empty pages to be
> > written
> > > from
> > > > > MessageColumnIO
> > > > > - PARQUET-1544: Possible over-shading of modules
> > > > >
> > > > > The following change has been reverted so it is not part of any
> > public
> > > > > release:
> > > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > > > >
> > > > > Please download, verify, and test. The vote will be open for at
> > least
> > > 72
> > > > > hours.
> > > > >
> > > > > Thanks,
> > > > > Gabor
> > > > >
> > >
> >
> >
> >
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by Zoltan Ivanfi <zi...@cloudera.com.INVALID>.
Hi,
That error message is quite surprising, Maven should be able to find that
artifact without any manual configuration. It may have been some temporary
network issue with accessing the repository. Could you try it again? If it
keeps failing, Maven may have cached the unavailability of that artifact,
which can be cleaned by issuing
mvn dependency:purge-local-repository
-DmanualInclude=com.hadoop.gplcompression:hadoop-lzo
In case you still continue to get the same failure, you may be behind a
stale artifact cache or some other kind of proxy or that does not allow you
to access this artifact directly.
Br,
Zoltan
On Wed, Apr 3, 2019 at 2:14 PM XU Qinghui <qi...@gmail.com> wrote:
> Hello, Andy
>
> I think this dependency is in maven repository:
>
> https://mvnrepository.com/artifact/com.hadoop.gplcompression/hadoop-lzo/0.4.16
> Maybe you need to configure your `.m2/settings.xml` to have the repository,
> here's some doc from maven:
> https://maven.apache.org/guides/mini/guide-multiple-repositories.html
>
> Best wishes,
>
> Le mer. 3 avr. 2019 à 13:33, Andy Grove <An...@rms.com> a écrit :
>
> > I've been watching the RC process here and would like to help verify the
> > release, since I have co-workers wanting to upgrade to 1.11
> >
> > When I follow the instructions to verify the release I am running into an
> > issue resolving a dependency:
> >
> > Could not resolve dependencies for project
> > org.apache.parquet:parquet-thrift:jar:1.11.0: Could not find artifact
> > com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16
> >
> > I don't see anything in the README in the repo that explains how I should
> > configure maven to find this dependency. I don’t even see a direct
> > reference to this jar in maven, so would appreciate some guidance.
> >
> > Thanks,
> >
> > Andy,
> >
> >
> > On 3/21/19, 3:40 PM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
> >
> > CAUTION – UNVERIFIED EXTERNAL EMAIL
> >
> >
> > Hi Wes,
> >
> > Here is a list of steps (the first part is probably the same as for
> > parquet-cpp):
> >
> > 1. Download the files from
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=5WxAct62mkcX2OezQNaj15X9Q%2FQ3zJETYBNt2kKFyWk%3D&reserved=0
> > 2. Download the KEYS file from
> >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=yy4O7iXv8RZshf4qyarjyzPyIyNq4p3rtkjjNUWcIRY%3D&reserved=0
> > 3. Import the GPG keys by running `gpg --import KEYS`.
> > 4. Verify the signature by running `gpg --verify
> > apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature",
> the
> > warning about the key not being trusted can be ignored.
> > 5. Extract the source code: `tar xzf apache-parquet-1.11.0.tar.gz`.
> > 6. Change to the source directory: `cd apache-parquet-1.11.0`.
> > 7. Make sure that you have a thrift compiler and that you have the
> > correct
> > version by issuing `thrift -version`. It should print "Thrift version
> > 0.12.0". If you don't have it, you need to install it in a manner
> that
> > is
> > specific to your system (for example, `sudo apt install
> > thrift-compiler` or
> > `sudo yum install thrift-compiler`). If your package manager only
> > offers an
> > older version, you may still use that with the workaround shown
> below.
> > 8. Test the release by issuing `mvn clean verify`. If you have an
> older
> > thrift compiler, for example 0.9.3, you have to issue `mvn
> > -Dformat.thrift.version=0.9.3 -Dthrift.version=0.9.3 clean install`
> > instead. This may or may not work depending on how old your thrift
> > version
> > is (>= 0.9.0 should work). Running this command can take around 15
> > minutes
> > and should print "BUILD SUCCESS" at the end (followed by a few extra
> > lines).
> >
> > Br,
> >
> > Zoltan
> >
> > On Thu, Mar 21, 2019 at 7:58 PM Wes McKinney <we...@gmail.com>
> > wrote:
> >
> > > Are there any instructions written down about how to verify this
> > release?
> > >
> > > On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi
> > <zi...@cloudera.com.invalid>
> > > wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > > signature matches
> > > > git hash matches the git tag
> > > > source tarball matches the git tag
> > > > unit tests and integration tests pass
> > > >
> > > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <
> gabor@apache.org
> > >
> > > wrote:
> > > >
> > > > > Dear Parquet Users and Developers,
> > > > >
> > > > > I propose the following RC to be released as the official
> Apache
> > > > > Parquet 1.11.0 release:
> > > > >
> > > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > > *
> > > > >
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=MoO3FhiXqeXRt0jb0lvBqsNELuUUwKfjekX6DSvOr%2Fg%3D&reserved=0
> > > > >
> > > > > The release tarball, signature, and checksums are here:
> > > > > *
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=5WxAct62mkcX2OezQNaj15X9Q%2FQ3zJETYBNt2kKFyWk%3D&reserved=0
> > > > >
> > > > > You can find the KEYS file here:
> > > > > *
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fformatdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=e6W8VpBIil0xTYrzoJpNRR%2FCiKNClA87nxY3Y6UNmvM%3D&reserved=0
> > > <
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=yy4O7iXv8RZshf4qyarjyzPyIyNq4p3rtkjjNUWcIRY%3D&reserved=0
> > >
> > > > >
> > > > > Binary artifacts are staged in Nexus here:
> > > > > *
> > > > >
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=NAe4dRJv%2BNuFXnidnLD2Ljc7WuQxlryqbVZlwsNKLag%3D&reserved=0
> > > > >
> > > > > This release includes the following new features:
> > > > > - PARQUET-1201 - Column indexes
> > > > > - PARQUET-1253 - Support for new logical type representation
> > > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > > > - PARQUET-1388 - Nanosecond precision time and timestamp -
> > parquet-mr
> > > > >
> > > > > The release also includes bug fixes, including:
> > > > > - PARQUET-1472: Dictionary filter fails on
> FIXED_LEN_BYTE_ARRAY.
> > > > > - PARQUET-1510: Fix notEq for optional columns with null
> values.
> > > > > - PARQUET-1533: TestSnappy() throws OOM exception with
> > Parquet-1485
> > > change
> > > > > - PARQUET-1531: Page row count limit causes empty pages to be
> > written
> > > from
> > > > > MessageColumnIO
> > > > > - PARQUET-1544: Possible over-shading of modules
> > > > >
> > > > > The following change has been reverted so it is not part of any
> > public
> > > > > release:
> > > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > > > >
> > > > > Please download, verify, and test. The vote will be open for at
> > least
> > > 72
> > > > > hours.
> > > > >
> > > > > Thanks,
> > > > > Gabor
> > > > >
> > >
> >
> >
> >
>
Re: [VOTE] Release Apache Parquet 1.11.0 RC6
Posted by XU Qinghui <qi...@gmail.com>.
Hello, Andy
I think this dependency is in maven repository:
https://mvnrepository.com/artifact/com.hadoop.gplcompression/hadoop-lzo/0.4.16
Maybe you need to configure your `.m2/settings.xml` to have the repository,
here's some doc from maven:
https://maven.apache.org/guides/mini/guide-multiple-repositories.html
Best wishes,
Le mer. 3 avr. 2019 à 13:33, Andy Grove <An...@rms.com> a écrit :
> I've been watching the RC process here and would like to help verify the
> release, since I have co-workers wanting to upgrade to 1.11
>
> When I follow the instructions to verify the release I am running into an
> issue resolving a dependency:
>
> Could not resolve dependencies for project
> org.apache.parquet:parquet-thrift:jar:1.11.0: Could not find artifact
> com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16
>
> I don't see anything in the README in the repo that explains how I should
> configure maven to find this dependency. I don’t even see a direct
> reference to this jar in maven, so would appreciate some guidance.
>
> Thanks,
>
> Andy,
>
>
> On 3/21/19, 3:40 PM, "Zoltan Ivanfi" <zi...@cloudera.com.INVALID> wrote:
>
> CAUTION – UNVERIFIED EXTERNAL EMAIL
>
>
> Hi Wes,
>
> Here is a list of steps (the first part is probably the same as for
> parquet-cpp):
>
> 1. Download the files from
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=5WxAct62mkcX2OezQNaj15X9Q%2FQ3zJETYBNt2kKFyWk%3D&reserved=0
> 2. Download the KEYS file from
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=yy4O7iXv8RZshf4qyarjyzPyIyNq4p3rtkjjNUWcIRY%3D&reserved=0
> 3. Import the GPG keys by running `gpg --import KEYS`.
> 4. Verify the signature by running `gpg --verify
> apache-parquet-1.11.0.tar.gz.asc`. It should say "Good signature", the
> warning about the key not being trusted can be ignored.
> 5. Extract the source code: `tar xzf apache-parquet-1.11.0.tar.gz`.
> 6. Change to the source directory: `cd apache-parquet-1.11.0`.
> 7. Make sure that you have a thrift compiler and that you have the
> correct
> version by issuing `thrift -version`. It should print "Thrift version
> 0.12.0". If you don't have it, you need to install it in a manner that
> is
> specific to your system (for example, `sudo apt install
> thrift-compiler` or
> `sudo yum install thrift-compiler`). If your package manager only
> offers an
> older version, you may still use that with the workaround shown below.
> 8. Test the release by issuing `mvn clean verify`. If you have an older
> thrift compiler, for example 0.9.3, you have to issue `mvn
> -Dformat.thrift.version=0.9.3 -Dthrift.version=0.9.3 clean install`
> instead. This may or may not work depending on how old your thrift
> version
> is (>= 0.9.0 should work). Running this command can take around 15
> minutes
> and should print "BUILD SUCCESS" at the end (followed by a few extra
> lines).
>
> Br,
>
> Zoltan
>
> On Thu, Mar 21, 2019 at 7:58 PM Wes McKinney <we...@gmail.com>
> wrote:
>
> > Are there any instructions written down about how to verify this
> release?
> >
> > On Wed, Mar 20, 2019 at 8:50 AM Zoltan Ivanfi
> <zi...@cloudera.com.invalid>
> > wrote:
> > >
> > > +1 (binding)
> > >
> > > signature matches
> > > git hash matches the git tag
> > > source tarball matches the git tag
> > > unit tests and integration tests pass
> > >
> > > On Tue, Mar 19, 2019 at 3:00 PM Gabor Szadovszky <gabor@apache.org
> >
> > wrote:
> > >
> > > > Dear Parquet Users and Developers,
> > > >
> > > > I propose the following RC to be released as the official Apache
> > > > Parquet 1.11.0 release:
> > > >
> > > > The commit id is 9756b0e2b35437a09716707a81e2ac0c187112ed
> > > > * This corresponds to the tag: apache-parquet-1.11.0
> > > > *
> > > >
> > > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2F9756b0e2b35437a09716707a81e2ac0c187112ed&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=MoO3FhiXqeXRt0jb0lvBqsNELuUUwKfjekX6DSvOr%2Fg%3D&reserved=0
> > > >
> > > > The release tarball, signature, and checksums are here:
> > > > *
> > > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.11.0-rc6%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=5WxAct62mkcX2OezQNaj15X9Q%2FQ3zJETYBNt2kKFyWk%3D&reserved=0
> > > >
> > > > You can find the KEYS file here:
> > > > *
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fformatdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=e6W8VpBIil0xTYrzoJpNRR%2FCiKNClA87nxY3Y6UNmvM%3D&reserved=0
> > <
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2FKEYS&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=yy4O7iXv8RZshf4qyarjyzPyIyNq4p3rtkjjNUWcIRY%3D&reserved=0
> >
> > > >
> > > > Binary artifacts are staged in Nexus here:
> > > > *
> > > >
> > > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2Fparquet%2F1.11.0%2F&data=02%7C01%7CAndy.Grove%40rms.com%7C94c5ce21c66047bee55208d6ae45d8e2%7Cd43fb8a804da4990b86cc4ba9ba4511f%7C0%7C0%7C636888012357234811&sdata=NAe4dRJv%2BNuFXnidnLD2Ljc7WuQxlryqbVZlwsNKLag%3D&reserved=0
> > > >
> > > > This release includes the following new features:
> > > > - PARQUET-1201 - Column indexes
> > > > - PARQUET-1253 - Support for new logical type representation
> > > > - PARQUET-1381 - Add merge blocks command to parquet-tools
> > > > - PARQUET-1388 - Nanosecond precision time and timestamp -
> parquet-mr
> > > >
> > > > The release also includes bug fixes, including:
> > > > - PARQUET-1472: Dictionary filter fails on FIXED_LEN_BYTE_ARRAY.
> > > > - PARQUET-1510: Fix notEq for optional columns with null values.
> > > > - PARQUET-1533: TestSnappy() throws OOM exception with
> Parquet-1485
> > change
> > > > - PARQUET-1531: Page row count limit causes empty pages to be
> written
> > from
> > > > MessageColumnIO
> > > > - PARQUET-1544: Possible over-shading of modules
> > > >
> > > > The following change has been reverted so it is not part of any
> public
> > > > release:
> > > > - PARQUET-1381: Add merge blocks command to parquet-tools
> > > >
> > > > Please download, verify, and test. The vote will be open for at
> least
> > 72
> > > > hours.
> > > >
> > > > Thanks,
> > > > Gabor
> > > >
> >
>
>
>