You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Eduard Tudenhoefner <ed...@dremio.com> on 2022/06/02 06:11:47 UTC
[VOTE] Release Apache Iceberg 0.13.2 RC1
Hi Everyone,
I propose that we release the following RC as the official Apache Iceberg
0.13.2 release.
The commit ID is *0784d64a659abd4fdaa82cdb599a250a7514facf*
- This corresponds to the tag: *apache-iceberg-0.13.2-rc1*
- https://github.com/apache/iceberg/commits/apache-iceberg-0.13.2-rc1
-
https://github.com/apache/iceberg/tree/0784d64a659abd4fdaa82cdb599a250a7514facf
The release tarball, signature, and checksums are here:
-
https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.2-rc1
You can find the KEYS file here:
- https://dist.apache.org/repos/dist/dev/iceberg/KEYS
Convenience binary artifacts are staged on Nexus. The Maven repository URL
is:
-
https://repository.apache.org/content/repositories/orgapacheiceberg-1088/
Please download, verify, and test.
Please vote in the next 72 hours.
[ ] +1 Release this as Apache Iceberg <VERSION>
[ ] +0
[ ] -1 Do not release this because...
Re: [VOTE] Release Apache Iceberg 0.13.2 RC1
Posted by Eduard Tudenhoefner <ed...@dremio.com>.
+1 (non-binding)
* Verified signatures, checksums, and license checks
* verified build and tests pass
* verified Nessie <https://github.com/projectnessie/nessie/pull/4472> /
Trino <https://github.com/trinodb/trino/pull/12816> / Presto
<https://github.com/prestodb/presto/pull/17866> Iceberg tests pass
Eduard
On Fri, Jun 10, 2022 at 9:14 PM Jahagirdar, Amogh
<ja...@amazon.com.invalid> wrote:
> +1 non-binding
>
>
>
> - Validated signature, checksum and license checks
> - Validated all unit tests passed
> - Validated AWS integration tests passed
>
>
>
> Thanks,
>
>
>
> Amogh Jahagirdar
>
>
>
>
>
> *From: *Jack Ye <ye...@gmail.com>
> *Reply-To: *"dev@iceberg.apache.org" <de...@iceberg.apache.org>
> *Date: *Friday, June 10, 2022 at 10:46 AM
> *To: *Iceberg Dev List <de...@iceberg.apache.org>
> *Subject: *RE: [EXTERNAL][VOTE] Release Apache Iceberg 0.13.2 RC1
>
>
>
> *CAUTION*: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> +1 (binding)
>
>
>
> - Ran license checks, signature and checksum
>
> - Ran all unit tests
>
> - Ran AWS integration tests and internal release tests
>
>
>
> Best,
>
> Jack Ye
>
>
>
>
>
> On Thu, Jun 9, 2022 at 4:41 PM Ryan Blue <bl...@tabular.io> wrote:
>
> +1 (binding)
>
>
>
> * Ran license checks
>
> * Checked signature and checksum
>
> * Built/tested
>
>
>
> Looks good to me! Thanks, Eduard!
>
>
>
> For the detached ref issue, that's on purpose. The script doesn't add
> version.txt to the base branch because we don't want to add it to master
> only to need to remove it. It also doesn't affect the tag, which is based
> on a commit in the 0.13.x branch. Once 0.13.2 is out, we'll update the
> branch.
>
>
>
> Ryan
>
>
>
> On Tue, Jun 7, 2022 at 3:56 AM Peter Vary <pv...@cloudera.com.invalid>
> wrote:
>
> +1 (I am a committer but not a PMC, so I am not sure if this is binding,
> or not :D)
>
>
>
> Verified the code, sums and build.
>
> I was able to run the hive3 tests without issues.
>
>
>
> Thanks,
>
> Peter
>
>
>
> On 2022. Jun 7., at 4:55, John Zhuge <jz...@apache.org> wrote:
>
>
>
> +1 (non-binding)
>
> Verified sigs, sums, license, build and test.
>
>
>
> Tested simple table ops with Spark 3.2.
>
>
>
> On Mon, Jun 6, 2022 at 5:39 PM Szehon Ho <sz...@gmail.com> wrote:
>
> +1 (non-binding)
>
>
>
> 1. Verified signatures
> 2. Verified checksums
> 3. RAT checks
> 4. Build and test
> 5. Tested with Spark 3.2, create a table and run a few queries
>
> Thanks
>
> Szehon
>
>
>
> On Mon, Jun 6, 2022 at 10:46 AM Daniel Weeks <da...@gmail.com>
> wrote:
>
> +1 (binding)
>
>
>
> verified sigs/sums/license/build/tests
>
>
>
> As for the detached commit, I believe I commented on this in a prior
> release and the parent commit is the head of the 0.13.x branch and the
> detached commit is just the version bump, so I'm ok with it, but it sure
> would be nice if that wasn't detached.
>
>
>
> -Dan
>
>
>
> On Sun, Jun 5, 2022 at 10:27 PM Kyle Bendickson <ky...@tabular.io> wrote:
>
> Update:
>
>
>
> Running the test suite in IntelliJ that was (and is) having consistent
> test failures via CLI, the issue seems to be resolved.
>
> So I do think it is indeed a local JVM set up issue.
>
>
>
> Investigating the differences now, but the class in question is
> *org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine*
>
>
>
> It seems to be caused by a NoClassDefFoundError, specifically for
> org.xerial.snappy.Snappy. It also happens for ORC, but not for parquet.
>
>
>
> Included is a sample output:
>
> ```
>
> java.lang.NoClassDefFoundError: Could not initialize class
> org.xerial.snappy.Snappy
> at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:99)
> ~[snappy-java-1.1.8.jar:1.1.8]
> at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:91)
> ~[snappy-java-1.1.8.jar:1.1.8]
> at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:81)
> ~[snappy-java-1.1.8.jar:1.1.8]
> at
> org.apache.tez.common.TezUtils.createByteStringFromConf(TezUtils.java:81)
> ~[tez-api-0.10.1.jar:0.10.1]
>
> ```
>
>
>
> Apologies for speaking too soon. *I'm now +0 [non-binding] *provided we
> fix the 0.13.x branch and associated commitId to not be in a detached
> state. The tag *apache-iceberg-0.13.2-rc1 *works just fine, but the
> 0.13.x branch doesn't have the commit ID in question. Not sure if that's a
> major concern or not.
>
>
>
> Cheers,
>
> Kyle
>
>
>
> On Sun, Jun 5, 2022 at 11:51 AM Kyle Bendickson <ky...@tabular.io> wrote:
>
> Thanks Eduard!
>
>
>
> I have:
>
> - verified the signature
>
> - verified the checksum in the file given as well as of the artifact
>
> - ran all unit tests on Java 11, all passed
>
> - ran all unit tests on Java 8, some hive-3 tests consistently fail (I do
> notice they passed on Github - but the tests which fail are consistent
> despite giving the JVM more memory and checking for OOM)
>
> - ran a simple smoke test suite of CRUD on namespaces and v1 and v2 tables
> with Spark (3.2, 3.1) and Flink (1.13 and 1.14).
>
> - ran some upsert related tests on Flink 1.13 and 1.14 (1.12 is provided a
> deprecation notice)
>
>
>
> *Problems:*
>
> I did notice that the *given commit ID is considered unattached (and I
> wasn't able to check it out).* I am running my tests by using the
> provided JAR with engines and then running unit tests locally for the
> commit just prior (with commit ID
> *fae977a9f0a79266a04647b0df2ab540cf0dcff4*).
>
>
>
> Not sure if this is a huge issue, but outside of this unattached commit,
> my only concern is the `iceberg-hive3` failing tests, but as they passed in
> CI it's possibly an issue with my local setup locally.
>
>
>
> Running hive-3 test suite alone, the same tests failed multiple times but
> again might be something to do with my computer / JVM configuration.
>
>
>
> *I am -1 (non-binding)*, primarily based on the detached commit (as I had
> quite a good bit of trouble trying to fetch it through my normal processes)
> as well as the failing hive3 tests (though that's not exactly within my
> area of expertise).
>
>
>
> If the hive3 test failures are only something that occurs for me, then if
> we fix the "Add version.txt commit" in branch 0.3.x such that when I fetch
> branch 0.3.x it's present, I'd be +1. Unfortunately, I can't help with
> cleaning up with the release branch outside of advising somebody else (if
> desired), but I'm happy to help with that.
>
>
>
> The hive3 test failures for me seem to be OOM related, but I raised my
>
>
>
> Find attached a picture of the detached commit ID,
> *0784d64a659abd4fdaa82cdb599a250a7514facf*, per Github.
>
>
>
> *Error! Filename not specified.*
>
>
>
> Example test failures
>
> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine >
> testCBOWithSelectedColumnsOverlapJoin[fileFormat=AVRO, engine=tez,
> catalog=HIVE_CATALOG, isVectorized=false] FAILED
> java.lang.IllegalArgumentException: Failed to execute Hive query
> 'SELECT c.first_name, o.order_id FROM default.orders o JOIN
> default.customers c ON o.customer_id = c.customer_id ORDER BY o.order_id
> DESC': Error while processing statement: FAILED: Execution Error, return
> code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
> at
> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:152)
> at
> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine.testCBOWithSelectedColumnsOverlapJoin(TestHiveIcebergStorageHandlerWithEngine.java:236)
>
> Caused by:
> org.apache.hive.service.cli.HiveSQLException: Error while
> processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:263)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:510)
> at
> org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:267)
> at
> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:139)
> ... 1 more
>
>
>
> Thanks you for working on this,
>
> Kyle
>
>
>
>
>
>
>
>
>
> On Wed, Jun 1, 2022 at 11:12 PM Eduard Tudenhoefner <ed...@dremio.com>
> wrote:
>
> Hi Everyone,
>
> I propose that we release the following RC as the official Apache Iceberg
> 0.13.2 release.
>
> The commit ID is *0784d64a659abd4fdaa82cdb599a250a7514facf*
>
>
>
> - This corresponds to the tag: *apache-iceberg-0.13.2-rc1*
> - https://github.com/apache/iceberg/commits/apache-iceberg-0.13.2-rc1
> -
> https://github.com/apache/iceberg/tree/0784d64a659abd4fdaa82cdb599a250a7514facf
>
>
> The release tarball, signature, and checksums are here:
>
> -
> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.2-rc1
>
>
> You can find the KEYS file here:
>
> - https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>
>
> Convenience binary artifacts are staged on Nexus. The Maven repository URL
> is:
>
> -
> https://repository.apache.org/content/repositories/orgapacheiceberg-1088/
>
>
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as Apache Iceberg <VERSION>
> [ ] +0
> [ ] -1 Do not release this because...
>
>
>
>
> --
>
> John Zhuge
>
>
>
>
>
>
> --
>
> Ryan Blue
>
> Tabular
>
>
Re: [VOTE] Release Apache Iceberg 0.13.2 RC1
Posted by "Jahagirdar, Amogh" <ja...@amazon.com.INVALID>.
+1 non-binding
* Validated signature, checksum and license checks
* Validated all unit tests passed
* Validated AWS integration tests passed
Thanks,
Amogh Jahagirdar
From: Jack Ye <ye...@gmail.com>
Reply-To: "dev@iceberg.apache.org" <de...@iceberg.apache.org>
Date: Friday, June 10, 2022 at 10:46 AM
To: Iceberg Dev List <de...@iceberg.apache.org>
Subject: RE: [EXTERNAL][VOTE] Release Apache Iceberg 0.13.2 RC1
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
+1 (binding)
- Ran license checks, signature and checksum
- Ran all unit tests
- Ran AWS integration tests and internal release tests
Best,
Jack Ye
On Thu, Jun 9, 2022 at 4:41 PM Ryan Blue <bl...@tabular.io>> wrote:
+1 (binding)
* Ran license checks
* Checked signature and checksum
* Built/tested
Looks good to me! Thanks, Eduard!
For the detached ref issue, that's on purpose. The script doesn't add version.txt to the base branch because we don't want to add it to master only to need to remove it. It also doesn't affect the tag, which is based on a commit in the 0.13.x branch. Once 0.13.2 is out, we'll update the branch.
Ryan
On Tue, Jun 7, 2022 at 3:56 AM Peter Vary <pv...@cloudera.com.invalid> wrote:
+1 (I am a committer but not a PMC, so I am not sure if this is binding, or not :D)
Verified the code, sums and build.
I was able to run the hive3 tests without issues.
Thanks,
Peter
On 2022. Jun 7., at 4:55, John Zhuge <jz...@apache.org>> wrote:
+1 (non-binding)
Verified sigs, sums, license, build and test.
Tested simple table ops with Spark 3.2.
On Mon, Jun 6, 2022 at 5:39 PM Szehon Ho <sz...@gmail.com>> wrote:
+1 (non-binding)
1. Verified signatures
2. Verified checksums
3. RAT checks
4. Build and test
5. Tested with Spark 3.2, create a table and run a few queries
Thanks
Szehon
On Mon, Jun 6, 2022 at 10:46 AM Daniel Weeks <da...@gmail.com>> wrote:
+1 (binding)
verified sigs/sums/license/build/tests
As for the detached commit, I believe I commented on this in a prior release and the parent commit is the head of the 0.13.x branch and the detached commit is just the version bump, so I'm ok with it, but it sure would be nice if that wasn't detached.
-Dan
On Sun, Jun 5, 2022 at 10:27 PM Kyle Bendickson <ky...@tabular.io>> wrote:
Update:
Running the test suite in IntelliJ that was (and is) having consistent test failures via CLI, the issue seems to be resolved.
So I do think it is indeed a local JVM set up issue.
Investigating the differences now, but the class in question is org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine
It seems to be caused by a NoClassDefFoundError, specifically for org.xerial.snappy.Snappy. It also happens for ORC, but not for parquet.
Included is a sample output:
```
java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:99) ~[snappy-java-1.1.8.jar:1.1.8]
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:91) ~[snappy-java-1.1.8.jar:1.1.8]
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:81) ~[snappy-java-1.1.8.jar:1.1.8]
at org.apache.tez.common.TezUtils.createByteStringFromConf(TezUtils.java:81) ~[tez-api-0.10.1.jar:0.10.1]
```
Apologies for speaking too soon. I'm now +0 [non-binding] provided we fix the 0.13.x branch and associated commitId to not be in a detached state. The tag apache-iceberg-0.13.2-rc1 works just fine, but the 0.13.x branch doesn't have the commit ID in question. Not sure if that's a major concern or not.
Cheers,
Kyle
On Sun, Jun 5, 2022 at 11:51 AM Kyle Bendickson <ky...@tabular.io>> wrote:
Thanks Eduard!
I have:
- verified the signature
- verified the checksum in the file given as well as of the artifact
- ran all unit tests on Java 11, all passed
- ran all unit tests on Java 8, some hive-3 tests consistently fail (I do notice they passed on Github - but the tests which fail are consistent despite giving the JVM more memory and checking for OOM)
- ran a simple smoke test suite of CRUD on namespaces and v1 and v2 tables with Spark (3.2, 3.1) and Flink (1.13 and 1.14).
- ran some upsert related tests on Flink 1.13 and 1.14 (1.12 is provided a deprecation notice)
Problems:
I did notice that the given commit ID is considered unattached (and I wasn't able to check it out). I am running my tests by using the provided JAR with engines and then running unit tests locally for the commit just prior (with commit ID fae977a9f0a79266a04647b0df2ab540cf0dcff4).
Not sure if this is a huge issue, but outside of this unattached commit, my only concern is the `iceberg-hive3` failing tests, but as they passed in CI it's possibly an issue with my local setup locally.
Running hive-3 test suite alone, the same tests failed multiple times but again might be something to do with my computer / JVM configuration.
I am -1 (non-binding), primarily based on the detached commit (as I had quite a good bit of trouble trying to fetch it through my normal processes) as well as the failing hive3 tests (though that's not exactly within my area of expertise).
If the hive3 test failures are only something that occurs for me, then if we fix the "Add version.txt commit" in branch 0.3.x such that when I fetch branch 0.3.x it's present, I'd be +1. Unfortunately, I can't help with cleaning up with the release branch outside of advising somebody else (if desired), but I'm happy to help with that.
The hive3 test failures for me seem to be OOM related, but I raised my
Find attached a picture of the detached commit ID, 0784d64a659abd4fdaa82cdb599a250a7514facf, per Github.
Error! Filename not specified.
Example test failures
org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine > testCBOWithSelectedColumnsOverlapJoin[fileFormat=AVRO, engine=tez, catalog=HIVE_CATALOG, isVectorized=false] FAILED
java.lang.IllegalArgumentException: Failed to execute Hive query 'SELECT c.first_name, o.order_id FROM default.orders o JOIN default.customers c ON o.customer_id = c.customer_id ORDER BY o.order_id DESC': Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
at org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:152)
at org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine.testCBOWithSelectedColumnsOverlapJoin(TestHiveIcebergStorageHandlerWithEngine.java:236)
Caused by:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:263)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:510)
at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:267)
at org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:139)
... 1 more
Thanks you for working on this,
Kyle
On Wed, Jun 1, 2022 at 11:12 PM Eduard Tudenhoefner <ed...@dremio.com>> wrote:
Hi Everyone,
I propose that we release the following RC as the official Apache Iceberg 0.13.2 release.
The commit ID is 0784d64a659abd4fdaa82cdb599a250a7514facf
* This corresponds to the tag: apache-iceberg-0.13.2-rc1
* https://github.com/apache/iceberg/commits/apache-iceberg-0.13.2-rc1
* https://github.com/apache/iceberg/tree/0784d64a659abd4fdaa82cdb599a250a7514facf
The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.2-rc1
You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/iceberg/KEYS
Convenience binary artifacts are staged on Nexus. The Maven repository URL is:
* https://repository.apache.org/content/repositories/orgapacheiceberg-1088/
Please download, verify, and test.
Please vote in the next 72 hours.
[ ] +1 Release this as Apache Iceberg <VERSION>
[ ] +0
[ ] -1 Do not release this because...
--
John Zhuge
--
Ryan Blue
Tabular
Re: [VOTE] Release Apache Iceberg 0.13.2 RC1
Posted by Jack Ye <ye...@gmail.com>.
+1 (binding)
- Ran license checks, signature and checksum
- Ran all unit tests
- Ran AWS integration tests and internal release tests
Best,
Jack Ye
On Thu, Jun 9, 2022 at 4:41 PM Ryan Blue <bl...@tabular.io> wrote:
> +1 (binding)
>
> * Ran license checks
> * Checked signature and checksum
> * Built/tested
>
> Looks good to me! Thanks, Eduard!
>
> For the detached ref issue, that's on purpose. The script doesn't add
> version.txt to the base branch because we don't want to add it to master
> only to need to remove it. It also doesn't affect the tag, which is based
> on a commit in the 0.13.x branch. Once 0.13.2 is out, we'll update the
> branch.
>
> Ryan
>
> On Tue, Jun 7, 2022 at 3:56 AM Peter Vary <pv...@cloudera.com.invalid>
> wrote:
>
>> +1 (I am a committer but not a PMC, so I am not sure if this is binding,
>> or not :D)
>>
>> Verified the code, sums and build.
>> I was able to run the hive3 tests without issues.
>>
>> Thanks,
>> Peter
>>
>> On 2022. Jun 7., at 4:55, John Zhuge <jz...@apache.org> wrote:
>>
>> +1 (non-binding)
>>
>> Verified sigs, sums, license, build and test.
>>
>> Tested simple table ops with Spark 3.2.
>>
>> On Mon, Jun 6, 2022 at 5:39 PM Szehon Ho <sz...@gmail.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>>
>>> 1. Verified signatures
>>> 2. Verified checksums
>>> 3. RAT checks
>>> 4. Build and test
>>> 5. Tested with Spark 3.2, create a table and run a few queries
>>>
>>> Thanks
>>> Szehon
>>>
>>> On Mon, Jun 6, 2022 at 10:46 AM Daniel Weeks <da...@gmail.com>
>>> wrote:
>>>
>>>> +1 (binding)
>>>>
>>>> verified sigs/sums/license/build/tests
>>>>
>>>> As for the detached commit, I believe I commented on this in a prior
>>>> release and the parent commit is the head of the 0.13.x branch and the
>>>> detached commit is just the version bump, so I'm ok with it, but it sure
>>>> would be nice if that wasn't detached.
>>>>
>>>> -Dan
>>>>
>>>> On Sun, Jun 5, 2022 at 10:27 PM Kyle Bendickson <ky...@tabular.io>
>>>> wrote:
>>>>
>>>>> Update:
>>>>>
>>>>> Running the test suite in IntelliJ that was (and is) having consistent
>>>>> test failures via CLI, the issue seems to be resolved.
>>>>> So I do think it is indeed a local JVM set up issue.
>>>>>
>>>>> Investigating the differences now, but the class in question is
>>>>> *org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine*
>>>>>
>>>>> It seems to be caused by a NoClassDefFoundError, specifically for
>>>>> org.xerial.snappy.Snappy. It also happens for ORC, but not for parquet.
>>>>>
>>>>> Included is a sample output:
>>>>> ```
>>>>> java.lang.NoClassDefFoundError: Could not initialize class
>>>>> org.xerial.snappy.Snappy
>>>>> at
>>>>> org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:99)
>>>>> ~[snappy-java-1.1.8.jar:1.1.8]
>>>>> at
>>>>> org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:91)
>>>>> ~[snappy-java-1.1.8.jar:1.1.8]
>>>>> at
>>>>> org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:81)
>>>>> ~[snappy-java-1.1.8.jar:1.1.8]
>>>>> at
>>>>> org.apache.tez.common.TezUtils.createByteStringFromConf(TezUtils.java:81)
>>>>> ~[tez-api-0.10.1.jar:0.10.1]
>>>>> ```
>>>>>
>>>>> Apologies for speaking too soon. *I'm now +0 [non-binding] *provided
>>>>> we fix the 0.13.x branch and associated commitId to not be in a detached
>>>>> state. The tag *apache-iceberg-0.13.2-rc1 *works just fine, but the
>>>>> 0.13.x branch doesn't have the commit ID in question. Not sure if that's a
>>>>> major concern or not.
>>>>>
>>>>> Cheers,
>>>>> Kyle
>>>>>
>>>>> On Sun, Jun 5, 2022 at 11:51 AM Kyle Bendickson <ky...@tabular.io>
>>>>> wrote:
>>>>>
>>>>>> Thanks Eduard!
>>>>>>
>>>>>> I have:
>>>>>> - verified the signature
>>>>>> - verified the checksum in the file given as well as of the artifact
>>>>>> - ran all unit tests on Java 11, all passed
>>>>>> - ran all unit tests on Java 8, some hive-3 tests consistently fail
>>>>>> (I do notice they passed on Github - but the tests which fail are
>>>>>> consistent despite giving the JVM more memory and checking for OOM)
>>>>>> - ran a simple smoke test suite of CRUD on namespaces and v1 and v2
>>>>>> tables with Spark (3.2, 3.1) and Flink (1.13 and 1.14).
>>>>>> - ran some upsert related tests on Flink 1.13 and 1.14 (1.12 is
>>>>>> provided a deprecation notice)
>>>>>>
>>>>>> *Problems:*
>>>>>> I did notice that the *given commit ID is considered unattached (and
>>>>>> I wasn't able to check it out).* I am running my tests by using the
>>>>>> provided JAR with engines and then running unit tests locally for the
>>>>>> commit just prior (with commit ID
>>>>>> *fae977a9f0a79266a04647b0df2ab540cf0dcff4*).
>>>>>>
>>>>>> Not sure if this is a huge issue, but outside of this unattached
>>>>>> commit, my only concern is the `iceberg-hive3` failing tests, but as they
>>>>>> passed in CI it's possibly an issue with my local setup locally.
>>>>>>
>>>>>> Running hive-3 test suite alone, the same tests failed multiple times
>>>>>> but again might be something to do with my computer / JVM configuration.
>>>>>>
>>>>>> *I am -1 (non-binding)*, primarily based on the detached commit (as
>>>>>> I had quite a good bit of trouble trying to fetch it through my normal
>>>>>> processes) as well as the failing hive3 tests (though that's not exactly
>>>>>> within my area of expertise).
>>>>>>
>>>>>> If the hive3 test failures are only something that occurs for me,
>>>>>> then if we fix the "Add version.txt commit" in branch 0.3.x such that when
>>>>>> I fetch branch 0.3.x it's present, I'd be +1. Unfortunately, I can't help
>>>>>> with cleaning up with the release branch outside of advising somebody else
>>>>>> (if desired), but I'm happy to help with that.
>>>>>>
>>>>>> The hive3 test failures for me seem to be OOM related, but I raised
>>>>>> my
>>>>>>
>>>>>> Find attached a picture of the detached commit ID,
>>>>>> *0784d64a659abd4fdaa82cdb599a250a7514facf*, per Github.
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>> Example test failures
>>>>>> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine >
>>>>>> testCBOWithSelectedColumnsOverlapJoin[fileFormat=AVRO, engine=tez,
>>>>>> catalog=HIVE_CATALOG, isVectorized=false] FAILED
>>>>>> java.lang.IllegalArgumentException: Failed to execute Hive query
>>>>>> 'SELECT c.first_name, o.order_id FROM default.orders o JOIN
>>>>>> default.customers c ON o.customer_id = c.customer_id ORDER BY o.order_id
>>>>>> DESC': Error while processing statement: FAILED: Execution Error, return
>>>>>> code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
>>>>>> at
>>>>>> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:152)
>>>>>> at
>>>>>> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine.testCBOWithSelectedColumnsOverlapJoin(TestHiveIcebergStorageHandlerWithEngine.java:236)
>>>>>>
>>>>>> Caused by:
>>>>>> org.apache.hive.service.cli.HiveSQLException: Error while
>>>>>> processing statement: FAILED: Execution Error, return code 1 from
>>>>>> org.apache.hadoop.hive.ql.exec.tez.TezTask
>>>>>> at
>>>>>> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
>>>>>> at
>>>>>> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226)
>>>>>> at
>>>>>> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:263)
>>>>>> at
>>>>>> org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
>>>>>> at
>>>>>> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
>>>>>> at
>>>>>> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:510)
>>>>>> at
>>>>>> org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:267)
>>>>>> at
>>>>>> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:139)
>>>>>> ... 1 more
>>>>>>
>>>>>> Thanks you for working on this,
>>>>>> Kyle
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 1, 2022 at 11:12 PM Eduard Tudenhoefner <
>>>>>> eduard@dremio.com> wrote:
>>>>>>
>>>>>>> Hi Everyone,
>>>>>>>
>>>>>>> I propose that we release the following RC as the official Apache
>>>>>>> Iceberg 0.13.2 release.
>>>>>>>
>>>>>>> The commit ID is *0784d64a659abd4fdaa82cdb599a250a7514facf*
>>>>>>>
>>>>>>>
>>>>>>> - This corresponds to the tag: *apache-iceberg-0.13.2-rc1*
>>>>>>> -
>>>>>>> https://github.com/apache/iceberg/commits/apache-iceberg-0.13.2-rc1
>>>>>>> -
>>>>>>> https://github.com/apache/iceberg/tree/0784d64a659abd4fdaa82cdb599a250a7514facf
>>>>>>>
>>>>>>>
>>>>>>> The release tarball, signature, and checksums are here:
>>>>>>>
>>>>>>> -
>>>>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.2-rc1
>>>>>>>
>>>>>>>
>>>>>>> You can find the KEYS file here:
>>>>>>>
>>>>>>> - https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>>>>>>
>>>>>>>
>>>>>>> Convenience binary artifacts are staged on Nexus. The Maven
>>>>>>> repository URL is:
>>>>>>>
>>>>>>> -
>>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1088/
>>>>>>>
>>>>>>>
>>>>>>> Please download, verify, and test.
>>>>>>>
>>>>>>> Please vote in the next 72 hours.
>>>>>>>
>>>>>>> [ ] +1 Release this as Apache Iceberg <VERSION>
>>>>>>> [ ] +0
>>>>>>> [ ] -1 Do not release this because...
>>>>>>>
>>>>>>
>>
>> --
>> John Zhuge
>>
>>
>>
>
> --
> Ryan Blue
> Tabular
>
Re: [VOTE] Release Apache Iceberg 0.13.2 RC1
Posted by Ryan Blue <bl...@tabular.io>.
+1 (binding)
* Ran license checks
* Checked signature and checksum
* Built/tested
Looks good to me! Thanks, Eduard!
For the detached ref issue, that's on purpose. The script doesn't add
version.txt to the base branch because we don't want to add it to master
only to need to remove it. It also doesn't affect the tag, which is based
on a commit in the 0.13.x branch. Once 0.13.2 is out, we'll update the
branch.
Ryan
On Tue, Jun 7, 2022 at 3:56 AM Peter Vary <pv...@cloudera.com.invalid>
wrote:
> +1 (I am a committer but not a PMC, so I am not sure if this is binding,
> or not :D)
>
> Verified the code, sums and build.
> I was able to run the hive3 tests without issues.
>
> Thanks,
> Peter
>
> On 2022. Jun 7., at 4:55, John Zhuge <jz...@apache.org> wrote:
>
> +1 (non-binding)
>
> Verified sigs, sums, license, build and test.
>
> Tested simple table ops with Spark 3.2.
>
> On Mon, Jun 6, 2022 at 5:39 PM Szehon Ho <sz...@gmail.com> wrote:
>
>> +1 (non-binding)
>>
>>
>> 1. Verified signatures
>> 2. Verified checksums
>> 3. RAT checks
>> 4. Build and test
>> 5. Tested with Spark 3.2, create a table and run a few queries
>>
>> Thanks
>> Szehon
>>
>> On Mon, Jun 6, 2022 at 10:46 AM Daniel Weeks <da...@gmail.com>
>> wrote:
>>
>>> +1 (binding)
>>>
>>> verified sigs/sums/license/build/tests
>>>
>>> As for the detached commit, I believe I commented on this in a prior
>>> release and the parent commit is the head of the 0.13.x branch and the
>>> detached commit is just the version bump, so I'm ok with it, but it sure
>>> would be nice if that wasn't detached.
>>>
>>> -Dan
>>>
>>> On Sun, Jun 5, 2022 at 10:27 PM Kyle Bendickson <ky...@tabular.io> wrote:
>>>
>>>> Update:
>>>>
>>>> Running the test suite in IntelliJ that was (and is) having consistent
>>>> test failures via CLI, the issue seems to be resolved.
>>>> So I do think it is indeed a local JVM set up issue.
>>>>
>>>> Investigating the differences now, but the class in question is
>>>> *org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine*
>>>>
>>>> It seems to be caused by a NoClassDefFoundError, specifically for
>>>> org.xerial.snappy.Snappy. It also happens for ORC, but not for parquet.
>>>>
>>>> Included is a sample output:
>>>> ```
>>>> java.lang.NoClassDefFoundError: Could not initialize class
>>>> org.xerial.snappy.Snappy
>>>> at
>>>> org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:99)
>>>> ~[snappy-java-1.1.8.jar:1.1.8]
>>>> at
>>>> org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:91)
>>>> ~[snappy-java-1.1.8.jar:1.1.8]
>>>> at
>>>> org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:81)
>>>> ~[snappy-java-1.1.8.jar:1.1.8]
>>>> at
>>>> org.apache.tez.common.TezUtils.createByteStringFromConf(TezUtils.java:81)
>>>> ~[tez-api-0.10.1.jar:0.10.1]
>>>> ```
>>>>
>>>> Apologies for speaking too soon. *I'm now +0 [non-binding] *provided
>>>> we fix the 0.13.x branch and associated commitId to not be in a detached
>>>> state. The tag *apache-iceberg-0.13.2-rc1 *works just fine, but the
>>>> 0.13.x branch doesn't have the commit ID in question. Not sure if that's a
>>>> major concern or not.
>>>>
>>>> Cheers,
>>>> Kyle
>>>>
>>>> On Sun, Jun 5, 2022 at 11:51 AM Kyle Bendickson <ky...@tabular.io>
>>>> wrote:
>>>>
>>>>> Thanks Eduard!
>>>>>
>>>>> I have:
>>>>> - verified the signature
>>>>> - verified the checksum in the file given as well as of the artifact
>>>>> - ran all unit tests on Java 11, all passed
>>>>> - ran all unit tests on Java 8, some hive-3 tests consistently fail (I
>>>>> do notice they passed on Github - but the tests which fail are consistent
>>>>> despite giving the JVM more memory and checking for OOM)
>>>>> - ran a simple smoke test suite of CRUD on namespaces and v1 and v2
>>>>> tables with Spark (3.2, 3.1) and Flink (1.13 and 1.14).
>>>>> - ran some upsert related tests on Flink 1.13 and 1.14 (1.12 is
>>>>> provided a deprecation notice)
>>>>>
>>>>> *Problems:*
>>>>> I did notice that the *given commit ID is considered unattached (and
>>>>> I wasn't able to check it out).* I am running my tests by using the
>>>>> provided JAR with engines and then running unit tests locally for the
>>>>> commit just prior (with commit ID
>>>>> *fae977a9f0a79266a04647b0df2ab540cf0dcff4*).
>>>>>
>>>>> Not sure if this is a huge issue, but outside of this unattached
>>>>> commit, my only concern is the `iceberg-hive3` failing tests, but as they
>>>>> passed in CI it's possibly an issue with my local setup locally.
>>>>>
>>>>> Running hive-3 test suite alone, the same tests failed multiple times
>>>>> but again might be something to do with my computer / JVM configuration.
>>>>>
>>>>> *I am -1 (non-binding)*, primarily based on the detached commit (as I
>>>>> had quite a good bit of trouble trying to fetch it through my normal
>>>>> processes) as well as the failing hive3 tests (though that's not exactly
>>>>> within my area of expertise).
>>>>>
>>>>> If the hive3 test failures are only something that occurs for me, then
>>>>> if we fix the "Add version.txt commit" in branch 0.3.x such that when I
>>>>> fetch branch 0.3.x it's present, I'd be +1. Unfortunately, I can't help
>>>>> with cleaning up with the release branch outside of advising somebody else
>>>>> (if desired), but I'm happy to help with that.
>>>>>
>>>>> The hive3 test failures for me seem to be OOM related, but I raised my
>>>>>
>>>>> Find attached a picture of the detached commit ID,
>>>>> *0784d64a659abd4fdaa82cdb599a250a7514facf*, per Github.
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> Example test failures
>>>>> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine >
>>>>> testCBOWithSelectedColumnsOverlapJoin[fileFormat=AVRO, engine=tez,
>>>>> catalog=HIVE_CATALOG, isVectorized=false] FAILED
>>>>> java.lang.IllegalArgumentException: Failed to execute Hive query
>>>>> 'SELECT c.first_name, o.order_id FROM default.orders o JOIN
>>>>> default.customers c ON o.customer_id = c.customer_id ORDER BY o.order_id
>>>>> DESC': Error while processing statement: FAILED: Execution Error, return
>>>>> code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
>>>>> at
>>>>> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:152)
>>>>> at
>>>>> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine.testCBOWithSelectedColumnsOverlapJoin(TestHiveIcebergStorageHandlerWithEngine.java:236)
>>>>>
>>>>> Caused by:
>>>>> org.apache.hive.service.cli.HiveSQLException: Error while
>>>>> processing statement: FAILED: Execution Error, return code 1 from
>>>>> org.apache.hadoop.hive.ql.exec.tez.TezTask
>>>>> at
>>>>> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
>>>>> at
>>>>> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226)
>>>>> at
>>>>> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:263)
>>>>> at
>>>>> org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
>>>>> at
>>>>> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
>>>>> at
>>>>> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:510)
>>>>> at
>>>>> org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:267)
>>>>> at
>>>>> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:139)
>>>>> ... 1 more
>>>>>
>>>>> Thanks you for working on this,
>>>>> Kyle
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jun 1, 2022 at 11:12 PM Eduard Tudenhoefner <ed...@dremio.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Everyone,
>>>>>>
>>>>>> I propose that we release the following RC as the official Apache
>>>>>> Iceberg 0.13.2 release.
>>>>>>
>>>>>> The commit ID is *0784d64a659abd4fdaa82cdb599a250a7514facf*
>>>>>>
>>>>>>
>>>>>> - This corresponds to the tag: *apache-iceberg-0.13.2-rc1*
>>>>>> -
>>>>>> https://github.com/apache/iceberg/commits/apache-iceberg-0.13.2-rc1
>>>>>> -
>>>>>> https://github.com/apache/iceberg/tree/0784d64a659abd4fdaa82cdb599a250a7514facf
>>>>>>
>>>>>>
>>>>>> The release tarball, signature, and checksums are here:
>>>>>>
>>>>>> -
>>>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.2-rc1
>>>>>>
>>>>>>
>>>>>> You can find the KEYS file here:
>>>>>>
>>>>>> - https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>>>>>
>>>>>>
>>>>>> Convenience binary artifacts are staged on Nexus. The Maven
>>>>>> repository URL is:
>>>>>>
>>>>>> -
>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1088/
>>>>>>
>>>>>>
>>>>>> Please download, verify, and test.
>>>>>>
>>>>>> Please vote in the next 72 hours.
>>>>>>
>>>>>> [ ] +1 Release this as Apache Iceberg <VERSION>
>>>>>> [ ] +0
>>>>>> [ ] -1 Do not release this because...
>>>>>>
>>>>>
>
> --
> John Zhuge
>
>
>
--
Ryan Blue
Tabular
Re: [VOTE] Release Apache Iceberg 0.13.2 RC1
Posted by Peter Vary <pv...@cloudera.com.INVALID>.
+1 (I am a committer but not a PMC, so I am not sure if this is binding, or not :D)
Verified the code, sums and build.
I was able to run the hive3 tests without issues.
Thanks,
Peter
> On 2022. Jun 7., at 4:55, John Zhuge <jz...@apache.org> wrote:
>
> +1 (non-binding)
>
> Verified sigs, sums, license, build and test.
>
> Tested simple table ops with Spark 3.2.
>
> On Mon, Jun 6, 2022 at 5:39 PM Szehon Ho <szehon.apache@gmail.com <ma...@gmail.com>> wrote:
> +1 (non-binding)
>
> Verified signatures
> Verified checksums
> RAT checks
> Build and test
> Tested with Spark 3.2, create a table and run a few queries
> Thanks
> Szehon
>
> On Mon, Jun 6, 2022 at 10:46 AM Daniel Weeks <daniel.c.weeks@gmail.com <ma...@gmail.com>> wrote:
> +1 (binding)
>
> verified sigs/sums/license/build/tests
>
> As for the detached commit, I believe I commented on this in a prior release and the parent commit is the head of the 0.13.x branch and the detached commit is just the version bump, so I'm ok with it, but it sure would be nice if that wasn't detached.
>
> -Dan
>
> On Sun, Jun 5, 2022 at 10:27 PM Kyle Bendickson <kyle@tabular.io <ma...@tabular.io>> wrote:
> Update:
>
> Running the test suite in IntelliJ that was (and is) having consistent test failures via CLI, the issue seems to be resolved.
> So I do think it is indeed a local JVM set up issue.
>
> Investigating the differences now, but the class in question is org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine
>
> It seems to be caused by a NoClassDefFoundError, specifically for org.xerial.snappy.Snappy. It also happens for ORC, but not for parquet.
>
> Included is a sample output:
> ```
> java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
> at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:99) ~[snappy-java-1.1.8.jar:1.1.8]
> at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:91) ~[snappy-java-1.1.8.jar:1.1.8]
> at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:81) ~[snappy-java-1.1.8.jar:1.1.8]
> at org.apache.tez.common.TezUtils.createByteStringFromConf(TezUtils.java:81) ~[tez-api-0.10.1.jar:0.10.1]
> ```
>
> Apologies for speaking too soon. I'm now +0 [non-binding] provided we fix the 0.13.x branch and associated commitId to not be in a detached state. The tag apache-iceberg-0.13.2-rc1 works just fine, but the 0.13.x branch doesn't have the commit ID in question. Not sure if that's a major concern or not.
>
> Cheers,
> Kyle
>
> On Sun, Jun 5, 2022 at 11:51 AM Kyle Bendickson <kyle@tabular.io <ma...@tabular.io>> wrote:
> Thanks Eduard!
>
> I have:
> - verified the signature
> - verified the checksum in the file given as well as of the artifact
> - ran all unit tests on Java 11, all passed
> - ran all unit tests on Java 8, some hive-3 tests consistently fail (I do notice they passed on Github - but the tests which fail are consistent despite giving the JVM more memory and checking for OOM)
> - ran a simple smoke test suite of CRUD on namespaces and v1 and v2 tables with Spark (3.2, 3.1) and Flink (1.13 and 1.14).
> - ran some upsert related tests on Flink 1.13 and 1.14 (1.12 is provided a deprecation notice)
>
> Problems:
> I did notice that the given commit ID is considered unattached (and I wasn't able to check it out). I am running my tests by using the provided JAR with engines and then running unit tests locally for the commit just prior (with commit ID fae977a9f0a79266a04647b0df2ab540cf0dcff4).
>
> Not sure if this is a huge issue, but outside of this unattached commit, my only concern is the `iceberg-hive3` failing tests, but as they passed in CI it's possibly an issue with my local setup locally.
>
> Running hive-3 test suite alone, the same tests failed multiple times but again might be something to do with my computer / JVM configuration.
>
> I am -1 (non-binding), primarily based on the detached commit (as I had quite a good bit of trouble trying to fetch it through my normal processes) as well as the failing hive3 tests (though that's not exactly within my area of expertise).
>
> If the hive3 test failures are only something that occurs for me, then if we fix the "Add version.txt commit" in branch 0.3.x such that when I fetch branch 0.3.x it's present, I'd be +1. Unfortunately, I can't help with cleaning up with the release branch outside of advising somebody else (if desired), but I'm happy to help with that.
>
> The hive3 test failures for me seem to be OOM related, but I raised my
>
> Find attached a picture of the detached commit ID, 0784d64a659abd4fdaa82cdb599a250a7514facf, per Github.
>
>
>
> Example test failures
> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine > testCBOWithSelectedColumnsOverlapJoin[fileFormat=AVRO, engine=tez, catalog=HIVE_CATALOG, isVectorized=false] FAILED
> java.lang.IllegalArgumentException: Failed to execute Hive query 'SELECT c.first_name, o.order_id FROM default.orders o JOIN default.customers c ON o.customer_id = c.customer_id ORDER BY o.order_id DESC': Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
> at org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:152)
> at org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine.testCBOWithSelectedColumnsOverlapJoin(TestHiveIcebergStorageHandlerWithEngine.java:236)
>
> Caused by:
> org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
> at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
> at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226)
> at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:263)
> at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
> at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
> at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:510)
> at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:267)
> at org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:139)
> ... 1 more
>
> Thanks you for working on this,
> Kyle
>
>
>
>
> On Wed, Jun 1, 2022 at 11:12 PM Eduard Tudenhoefner <eduard@dremio.com <ma...@dremio.com>> wrote:
> Hi Everyone,
>
> I propose that we release the following RC as the official Apache Iceberg 0.13.2 release.
>
> The commit ID is 0784d64a659abd4fdaa82cdb599a250a7514facf
>
> This corresponds to the tag: apache-iceberg-0.13.2-rc1
> https://github.com/apache/iceberg/commits/apache-iceberg-0.13.2-rc1 <https://github.com/apache/iceberg/commits/apache-iceberg-0.13.2-rc1>
> https://github.com/apache/iceberg/tree/0784d64a659abd4fdaa82cdb599a250a7514facf <https://github.com/apache/iceberg/tree/0784d64a659abd4fdaa82cdb599a250a7514facf>
> The release tarball, signature, and checksums are here:
> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.2-rc1 <https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.2-rc1>
> You can find the KEYS file here:
> https://dist.apache.org/repos/dist/dev/iceberg/KEYS <https://dist.apache.org/repos/dist/dev/iceberg/KEYS>
> Convenience binary artifacts are staged on Nexus. The Maven repository URL is:
> https://repository.apache.org/content/repositories/orgapacheiceberg-1088/ <https://repository.apache.org/content/repositories/orgapacheiceberg-1088/>
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as Apache Iceberg <VERSION>
> [ ] +0
> [ ] -1 Do not release this because...
>
>
> --
> John Zhuge
Re: [VOTE] Release Apache Iceberg 0.13.2 RC1
Posted by John Zhuge <jz...@apache.org>.
+1 (non-binding)
Verified sigs, sums, license, build and test.
Tested simple table ops with Spark 3.2.
On Mon, Jun 6, 2022 at 5:39 PM Szehon Ho <sz...@gmail.com> wrote:
> +1 (non-binding)
>
>
> 1. Verified signatures
> 2. Verified checksums
> 3. RAT checks
> 4. Build and test
> 5. Tested with Spark 3.2, create a table and run a few queries
>
> Thanks
> Szehon
>
> On Mon, Jun 6, 2022 at 10:46 AM Daniel Weeks <da...@gmail.com>
> wrote:
>
>> +1 (binding)
>>
>> verified sigs/sums/license/build/tests
>>
>> As for the detached commit, I believe I commented on this in a prior
>> release and the parent commit is the head of the 0.13.x branch and the
>> detached commit is just the version bump, so I'm ok with it, but it sure
>> would be nice if that wasn't detached.
>>
>> -Dan
>>
>> On Sun, Jun 5, 2022 at 10:27 PM Kyle Bendickson <ky...@tabular.io> wrote:
>>
>>> Update:
>>>
>>> Running the test suite in IntelliJ that was (and is) having consistent
>>> test failures via CLI, the issue seems to be resolved.
>>> So I do think it is indeed a local JVM set up issue.
>>>
>>> Investigating the differences now, but the class in question is
>>> *org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine*
>>>
>>> It seems to be caused by a NoClassDefFoundError, specifically for
>>> org.xerial.snappy.Snappy. It also happens for ORC, but not for parquet.
>>>
>>> Included is a sample output:
>>> ```
>>> java.lang.NoClassDefFoundError: Could not initialize class
>>> org.xerial.snappy.Snappy
>>> at
>>> org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:99)
>>> ~[snappy-java-1.1.8.jar:1.1.8]
>>> at
>>> org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:91)
>>> ~[snappy-java-1.1.8.jar:1.1.8]
>>> at
>>> org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:81)
>>> ~[snappy-java-1.1.8.jar:1.1.8]
>>> at
>>> org.apache.tez.common.TezUtils.createByteStringFromConf(TezUtils.java:81)
>>> ~[tez-api-0.10.1.jar:0.10.1]
>>> ```
>>>
>>> Apologies for speaking too soon. *I'm now +0 [non-binding] *provided we
>>> fix the 0.13.x branch and associated commitId to not be in a detached
>>> state. The tag *apache-iceberg-0.13.2-rc1 *works just fine, but the
>>> 0.13.x branch doesn't have the commit ID in question. Not sure if that's a
>>> major concern or not.
>>>
>>> Cheers,
>>> Kyle
>>>
>>> On Sun, Jun 5, 2022 at 11:51 AM Kyle Bendickson <ky...@tabular.io> wrote:
>>>
>>>> Thanks Eduard!
>>>>
>>>> I have:
>>>> - verified the signature
>>>> - verified the checksum in the file given as well as of the artifact
>>>> - ran all unit tests on Java 11, all passed
>>>> - ran all unit tests on Java 8, some hive-3 tests consistently fail (I
>>>> do notice they passed on Github - but the tests which fail are consistent
>>>> despite giving the JVM more memory and checking for OOM)
>>>> - ran a simple smoke test suite of CRUD on namespaces and v1 and v2
>>>> tables with Spark (3.2, 3.1) and Flink (1.13 and 1.14).
>>>> - ran some upsert related tests on Flink 1.13 and 1.14 (1.12 is
>>>> provided a deprecation notice)
>>>>
>>>> *Problems:*
>>>> I did notice that the *given commit ID is considered unattached (and I
>>>> wasn't able to check it out).* I am running my tests by using the
>>>> provided JAR with engines and then running unit tests locally for the
>>>> commit just prior (with commit ID
>>>> *fae977a9f0a79266a04647b0df2ab540cf0dcff4*).
>>>>
>>>> Not sure if this is a huge issue, but outside of this unattached
>>>> commit, my only concern is the `iceberg-hive3` failing tests, but as they
>>>> passed in CI it's possibly an issue with my local setup locally.
>>>>
>>>> Running hive-3 test suite alone, the same tests failed multiple times
>>>> but again might be something to do with my computer / JVM configuration.
>>>>
>>>> *I am -1 (non-binding)*, primarily based on the detached commit (as I
>>>> had quite a good bit of trouble trying to fetch it through my normal
>>>> processes) as well as the failing hive3 tests (though that's not exactly
>>>> within my area of expertise).
>>>>
>>>> If the hive3 test failures are only something that occurs for me, then
>>>> if we fix the "Add version.txt commit" in branch 0.3.x such that when I
>>>> fetch branch 0.3.x it's present, I'd be +1. Unfortunately, I can't help
>>>> with cleaning up with the release branch outside of advising somebody else
>>>> (if desired), but I'm happy to help with that.
>>>>
>>>> The hive3 test failures for me seem to be OOM related, but I raised my
>>>>
>>>> Find attached a picture of the detached commit ID,
>>>> *0784d64a659abd4fdaa82cdb599a250a7514facf*, per Github.
>>>>
>>>> [image: image.png]
>>>>
>>>> Example test failures
>>>> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine >
>>>> testCBOWithSelectedColumnsOverlapJoin[fileFormat=AVRO, engine=tez,
>>>> catalog=HIVE_CATALOG, isVectorized=false] FAILED
>>>> java.lang.IllegalArgumentException: Failed to execute Hive query
>>>> 'SELECT c.first_name, o.order_id FROM default.orders o JOIN
>>>> default.customers c ON o.customer_id = c.customer_id ORDER BY o.order_id
>>>> DESC': Error while processing statement: FAILED: Execution Error, return
>>>> code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
>>>> at
>>>> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:152)
>>>> at
>>>> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine.testCBOWithSelectedColumnsOverlapJoin(TestHiveIcebergStorageHandlerWithEngine.java:236)
>>>>
>>>> Caused by:
>>>> org.apache.hive.service.cli.HiveSQLException: Error while
>>>> processing statement: FAILED: Execution Error, return code 1 from
>>>> org.apache.hadoop.hive.ql.exec.tez.TezTask
>>>> at
>>>> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
>>>> at
>>>> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226)
>>>> at
>>>> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:263)
>>>> at
>>>> org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
>>>> at
>>>> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
>>>> at
>>>> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:510)
>>>> at
>>>> org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:267)
>>>> at
>>>> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:139)
>>>> ... 1 more
>>>>
>>>> Thanks you for working on this,
>>>> Kyle
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Jun 1, 2022 at 11:12 PM Eduard Tudenhoefner <ed...@dremio.com>
>>>> wrote:
>>>>
>>>>> Hi Everyone,
>>>>>
>>>>> I propose that we release the following RC as the official Apache
>>>>> Iceberg 0.13.2 release.
>>>>>
>>>>> The commit ID is *0784d64a659abd4fdaa82cdb599a250a7514facf*
>>>>>
>>>>>
>>>>> - This corresponds to the tag: *apache-iceberg-0.13.2-rc1*
>>>>> -
>>>>> https://github.com/apache/iceberg/commits/apache-iceberg-0.13.2-rc1
>>>>> -
>>>>> https://github.com/apache/iceberg/tree/0784d64a659abd4fdaa82cdb599a250a7514facf
>>>>>
>>>>>
>>>>> The release tarball, signature, and checksums are here:
>>>>>
>>>>> -
>>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.2-rc1
>>>>>
>>>>>
>>>>> You can find the KEYS file here:
>>>>>
>>>>> - https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>>>>
>>>>>
>>>>> Convenience binary artifacts are staged on Nexus. The Maven repository
>>>>> URL is:
>>>>>
>>>>> -
>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1088/
>>>>>
>>>>>
>>>>> Please download, verify, and test.
>>>>>
>>>>> Please vote in the next 72 hours.
>>>>>
>>>>> [ ] +1 Release this as Apache Iceberg <VERSION>
>>>>> [ ] +0
>>>>> [ ] -1 Do not release this because...
>>>>>
>>>>
--
John Zhuge
Re: [VOTE] Release Apache Iceberg 0.13.2 RC1
Posted by Szehon Ho <sz...@gmail.com>.
+1 (non-binding)
1. Verified signatures
2. Verified checksums
3. RAT checks
4. Build and test
5. Tested with Spark 3.2, create a table and run a few queries
Thanks
Szehon
On Mon, Jun 6, 2022 at 10:46 AM Daniel Weeks <da...@gmail.com>
wrote:
> +1 (binding)
>
> verified sigs/sums/license/build/tests
>
> As for the detached commit, I believe I commented on this in a prior
> release and the parent commit is the head of the 0.13.x branch and the
> detached commit is just the version bump, so I'm ok with it, but it sure
> would be nice if that wasn't detached.
>
> -Dan
>
> On Sun, Jun 5, 2022 at 10:27 PM Kyle Bendickson <ky...@tabular.io> wrote:
>
>> Update:
>>
>> Running the test suite in IntelliJ that was (and is) having consistent
>> test failures via CLI, the issue seems to be resolved.
>> So I do think it is indeed a local JVM set up issue.
>>
>> Investigating the differences now, but the class in question is
>> *org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine*
>>
>> It seems to be caused by a NoClassDefFoundError, specifically for
>> org.xerial.snappy.Snappy. It also happens for ORC, but not for parquet.
>>
>> Included is a sample output:
>> ```
>> java.lang.NoClassDefFoundError: Could not initialize class
>> org.xerial.snappy.Snappy
>> at
>> org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:99)
>> ~[snappy-java-1.1.8.jar:1.1.8]
>> at
>> org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:91)
>> ~[snappy-java-1.1.8.jar:1.1.8]
>> at
>> org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:81)
>> ~[snappy-java-1.1.8.jar:1.1.8]
>> at
>> org.apache.tez.common.TezUtils.createByteStringFromConf(TezUtils.java:81)
>> ~[tez-api-0.10.1.jar:0.10.1]
>> ```
>>
>> Apologies for speaking too soon. *I'm now +0 [non-binding] *provided we
>> fix the 0.13.x branch and associated commitId to not be in a detached
>> state. The tag *apache-iceberg-0.13.2-rc1 *works just fine, but the
>> 0.13.x branch doesn't have the commit ID in question. Not sure if that's a
>> major concern or not.
>>
>> Cheers,
>> Kyle
>>
>> On Sun, Jun 5, 2022 at 11:51 AM Kyle Bendickson <ky...@tabular.io> wrote:
>>
>>> Thanks Eduard!
>>>
>>> I have:
>>> - verified the signature
>>> - verified the checksum in the file given as well as of the artifact
>>> - ran all unit tests on Java 11, all passed
>>> - ran all unit tests on Java 8, some hive-3 tests consistently fail (I
>>> do notice they passed on Github - but the tests which fail are consistent
>>> despite giving the JVM more memory and checking for OOM)
>>> - ran a simple smoke test suite of CRUD on namespaces and v1 and v2
>>> tables with Spark (3.2, 3.1) and Flink (1.13 and 1.14).
>>> - ran some upsert related tests on Flink 1.13 and 1.14 (1.12 is provided
>>> a deprecation notice)
>>>
>>> *Problems:*
>>> I did notice that the *given commit ID is considered unattached (and I
>>> wasn't able to check it out).* I am running my tests by using the
>>> provided JAR with engines and then running unit tests locally for the
>>> commit just prior (with commit ID
>>> *fae977a9f0a79266a04647b0df2ab540cf0dcff4*).
>>>
>>> Not sure if this is a huge issue, but outside of this unattached commit,
>>> my only concern is the `iceberg-hive3` failing tests, but as they passed in
>>> CI it's possibly an issue with my local setup locally.
>>>
>>> Running hive-3 test suite alone, the same tests failed multiple times
>>> but again might be something to do with my computer / JVM configuration.
>>>
>>> *I am -1 (non-binding)*, primarily based on the detached commit (as I
>>> had quite a good bit of trouble trying to fetch it through my normal
>>> processes) as well as the failing hive3 tests (though that's not exactly
>>> within my area of expertise).
>>>
>>> If the hive3 test failures are only something that occurs for me, then
>>> if we fix the "Add version.txt commit" in branch 0.3.x such that when I
>>> fetch branch 0.3.x it's present, I'd be +1. Unfortunately, I can't help
>>> with cleaning up with the release branch outside of advising somebody else
>>> (if desired), but I'm happy to help with that.
>>>
>>> The hive3 test failures for me seem to be OOM related, but I raised my
>>>
>>> Find attached a picture of the detached commit ID,
>>> *0784d64a659abd4fdaa82cdb599a250a7514facf*, per Github.
>>>
>>> [image: image.png]
>>>
>>> Example test failures
>>> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine >
>>> testCBOWithSelectedColumnsOverlapJoin[fileFormat=AVRO, engine=tez,
>>> catalog=HIVE_CATALOG, isVectorized=false] FAILED
>>> java.lang.IllegalArgumentException: Failed to execute Hive query
>>> 'SELECT c.first_name, o.order_id FROM default.orders o JOIN
>>> default.customers c ON o.customer_id = c.customer_id ORDER BY o.order_id
>>> DESC': Error while processing statement: FAILED: Execution Error, return
>>> code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
>>> at
>>> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:152)
>>> at
>>> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine.testCBOWithSelectedColumnsOverlapJoin(TestHiveIcebergStorageHandlerWithEngine.java:236)
>>>
>>> Caused by:
>>> org.apache.hive.service.cli.HiveSQLException: Error while
>>> processing statement: FAILED: Execution Error, return code 1 from
>>> org.apache.hadoop.hive.ql.exec.tez.TezTask
>>> at
>>> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
>>> at
>>> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226)
>>> at
>>> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:263)
>>> at
>>> org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
>>> at
>>> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
>>> at
>>> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:510)
>>> at
>>> org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:267)
>>> at
>>> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:139)
>>> ... 1 more
>>>
>>> Thanks you for working on this,
>>> Kyle
>>>
>>>
>>>
>>>
>>> On Wed, Jun 1, 2022 at 11:12 PM Eduard Tudenhoefner <ed...@dremio.com>
>>> wrote:
>>>
>>>> Hi Everyone,
>>>>
>>>> I propose that we release the following RC as the official Apache
>>>> Iceberg 0.13.2 release.
>>>>
>>>> The commit ID is *0784d64a659abd4fdaa82cdb599a250a7514facf*
>>>>
>>>>
>>>> - This corresponds to the tag: *apache-iceberg-0.13.2-rc1*
>>>> -
>>>> https://github.com/apache/iceberg/commits/apache-iceberg-0.13.2-rc1
>>>> -
>>>> https://github.com/apache/iceberg/tree/0784d64a659abd4fdaa82cdb599a250a7514facf
>>>>
>>>>
>>>> The release tarball, signature, and checksums are here:
>>>>
>>>> -
>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.2-rc1
>>>>
>>>>
>>>> You can find the KEYS file here:
>>>>
>>>> - https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>>>
>>>>
>>>> Convenience binary artifacts are staged on Nexus. The Maven repository
>>>> URL is:
>>>>
>>>> -
>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1088/
>>>>
>>>>
>>>> Please download, verify, and test.
>>>>
>>>> Please vote in the next 72 hours.
>>>>
>>>> [ ] +1 Release this as Apache Iceberg <VERSION>
>>>> [ ] +0
>>>> [ ] -1 Do not release this because...
>>>>
>>>
Re: [VOTE] Release Apache Iceberg 0.13.2 RC1
Posted by Daniel Weeks <da...@gmail.com>.
+1 (binding)
verified sigs/sums/license/build/tests
As for the detached commit, I believe I commented on this in a prior
release and the parent commit is the head of the 0.13.x branch and the
detached commit is just the version bump, so I'm ok with it, but it sure
would be nice if that wasn't detached.
-Dan
On Sun, Jun 5, 2022 at 10:27 PM Kyle Bendickson <ky...@tabular.io> wrote:
> Update:
>
> Running the test suite in IntelliJ that was (and is) having consistent
> test failures via CLI, the issue seems to be resolved.
> So I do think it is indeed a local JVM set up issue.
>
> Investigating the differences now, but the class in question is
> *org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine*
>
> It seems to be caused by a NoClassDefFoundError, specifically for
> org.xerial.snappy.Snappy. It also happens for ORC, but not for parquet.
>
> Included is a sample output:
> ```
> java.lang.NoClassDefFoundError: Could not initialize class
> org.xerial.snappy.Snappy
> at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:99)
> ~[snappy-java-1.1.8.jar:1.1.8]
> at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:91)
> ~[snappy-java-1.1.8.jar:1.1.8]
> at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:81)
> ~[snappy-java-1.1.8.jar:1.1.8]
> at
> org.apache.tez.common.TezUtils.createByteStringFromConf(TezUtils.java:81)
> ~[tez-api-0.10.1.jar:0.10.1]
> ```
>
> Apologies for speaking too soon. *I'm now +0 [non-binding] *provided we
> fix the 0.13.x branch and associated commitId to not be in a detached
> state. The tag *apache-iceberg-0.13.2-rc1 *works just fine, but the
> 0.13.x branch doesn't have the commit ID in question. Not sure if that's a
> major concern or not.
>
> Cheers,
> Kyle
>
> On Sun, Jun 5, 2022 at 11:51 AM Kyle Bendickson <ky...@tabular.io> wrote:
>
>> Thanks Eduard!
>>
>> I have:
>> - verified the signature
>> - verified the checksum in the file given as well as of the artifact
>> - ran all unit tests on Java 11, all passed
>> - ran all unit tests on Java 8, some hive-3 tests consistently fail (I do
>> notice they passed on Github - but the tests which fail are consistent
>> despite giving the JVM more memory and checking for OOM)
>> - ran a simple smoke test suite of CRUD on namespaces and v1 and v2
>> tables with Spark (3.2, 3.1) and Flink (1.13 and 1.14).
>> - ran some upsert related tests on Flink 1.13 and 1.14 (1.12 is provided
>> a deprecation notice)
>>
>> *Problems:*
>> I did notice that the *given commit ID is considered unattached (and I
>> wasn't able to check it out).* I am running my tests by using the
>> provided JAR with engines and then running unit tests locally for the
>> commit just prior (with commit ID
>> *fae977a9f0a79266a04647b0df2ab540cf0dcff4*).
>>
>> Not sure if this is a huge issue, but outside of this unattached commit,
>> my only concern is the `iceberg-hive3` failing tests, but as they passed in
>> CI it's possibly an issue with my local setup locally.
>>
>> Running hive-3 test suite alone, the same tests failed multiple times but
>> again might be something to do with my computer / JVM configuration.
>>
>> *I am -1 (non-binding)*, primarily based on the detached commit (as I
>> had quite a good bit of trouble trying to fetch it through my normal
>> processes) as well as the failing hive3 tests (though that's not exactly
>> within my area of expertise).
>>
>> If the hive3 test failures are only something that occurs for me, then if
>> we fix the "Add version.txt commit" in branch 0.3.x such that when I fetch
>> branch 0.3.x it's present, I'd be +1. Unfortunately, I can't help with
>> cleaning up with the release branch outside of advising somebody else (if
>> desired), but I'm happy to help with that.
>>
>> The hive3 test failures for me seem to be OOM related, but I raised my
>>
>> Find attached a picture of the detached commit ID,
>> *0784d64a659abd4fdaa82cdb599a250a7514facf*, per Github.
>>
>> [image: image.png]
>>
>> Example test failures
>> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine >
>> testCBOWithSelectedColumnsOverlapJoin[fileFormat=AVRO, engine=tez,
>> catalog=HIVE_CATALOG, isVectorized=false] FAILED
>> java.lang.IllegalArgumentException: Failed to execute Hive query
>> 'SELECT c.first_name, o.order_id FROM default.orders o JOIN
>> default.customers c ON o.customer_id = c.customer_id ORDER BY o.order_id
>> DESC': Error while processing statement: FAILED: Execution Error, return
>> code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
>> at
>> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:152)
>> at
>> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine.testCBOWithSelectedColumnsOverlapJoin(TestHiveIcebergStorageHandlerWithEngine.java:236)
>>
>> Caused by:
>> org.apache.hive.service.cli.HiveSQLException: Error while
>> processing statement: FAILED: Execution Error, return code 1 from
>> org.apache.hadoop.hive.ql.exec.tez.TezTask
>> at
>> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
>> at
>> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226)
>> at
>> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:263)
>> at
>> org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
>> at
>> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
>> at
>> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:510)
>> at
>> org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:267)
>> at
>> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:139)
>> ... 1 more
>>
>> Thanks you for working on this,
>> Kyle
>>
>>
>>
>>
>> On Wed, Jun 1, 2022 at 11:12 PM Eduard Tudenhoefner <ed...@dremio.com>
>> wrote:
>>
>>> Hi Everyone,
>>>
>>> I propose that we release the following RC as the official Apache
>>> Iceberg 0.13.2 release.
>>>
>>> The commit ID is *0784d64a659abd4fdaa82cdb599a250a7514facf*
>>>
>>>
>>> - This corresponds to the tag: *apache-iceberg-0.13.2-rc1*
>>> - https://github.com/apache/iceberg/commits/apache-iceberg-0.13.2-rc1
>>> -
>>> https://github.com/apache/iceberg/tree/0784d64a659abd4fdaa82cdb599a250a7514facf
>>>
>>>
>>> The release tarball, signature, and checksums are here:
>>>
>>> -
>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.2-rc1
>>>
>>>
>>> You can find the KEYS file here:
>>>
>>> - https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>>
>>>
>>> Convenience binary artifacts are staged on Nexus. The Maven repository
>>> URL is:
>>>
>>> -
>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1088/
>>>
>>>
>>> Please download, verify, and test.
>>>
>>> Please vote in the next 72 hours.
>>>
>>> [ ] +1 Release this as Apache Iceberg <VERSION>
>>> [ ] +0
>>> [ ] -1 Do not release this because...
>>>
>>
Re: [VOTE] Release Apache Iceberg 0.13.2 RC1
Posted by Kyle Bendickson <ky...@tabular.io>.
Update:
Running the test suite in IntelliJ that was (and is) having consistent test
failures via CLI, the issue seems to be resolved.
So I do think it is indeed a local JVM set up issue.
Investigating the differences now, but the class in question is
*org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine*
It seems to be caused by a NoClassDefFoundError, specifically for
org.xerial.snappy.Snappy. It also happens for ORC, but not for parquet.
Included is a sample output:
```
java.lang.NoClassDefFoundError: Could not initialize class
org.xerial.snappy.Snappy
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:99)
~[snappy-java-1.1.8.jar:1.1.8]
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:91)
~[snappy-java-1.1.8.jar:1.1.8]
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:81)
~[snappy-java-1.1.8.jar:1.1.8]
at
org.apache.tez.common.TezUtils.createByteStringFromConf(TezUtils.java:81)
~[tez-api-0.10.1.jar:0.10.1]
```
Apologies for speaking too soon. *I'm now +0 [non-binding] *provided we fix
the 0.13.x branch and associated commitId to not be in a detached state.
The tag *apache-iceberg-0.13.2-rc1 *works just fine, but the 0.13.x branch
doesn't have the commit ID in question. Not sure if that's a major concern
or not.
Cheers,
Kyle
On Sun, Jun 5, 2022 at 11:51 AM Kyle Bendickson <ky...@tabular.io> wrote:
> Thanks Eduard!
>
> I have:
> - verified the signature
> - verified the checksum in the file given as well as of the artifact
> - ran all unit tests on Java 11, all passed
> - ran all unit tests on Java 8, some hive-3 tests consistently fail (I do
> notice they passed on Github - but the tests which fail are consistent
> despite giving the JVM more memory and checking for OOM)
> - ran a simple smoke test suite of CRUD on namespaces and v1 and v2 tables
> with Spark (3.2, 3.1) and Flink (1.13 and 1.14).
> - ran some upsert related tests on Flink 1.13 and 1.14 (1.12 is provided a
> deprecation notice)
>
> *Problems:*
> I did notice that the *given commit ID is considered unattached (and I
> wasn't able to check it out).* I am running my tests by using the
> provided JAR with engines and then running unit tests locally for the
> commit just prior (with commit ID
> *fae977a9f0a79266a04647b0df2ab540cf0dcff4*).
>
> Not sure if this is a huge issue, but outside of this unattached commit,
> my only concern is the `iceberg-hive3` failing tests, but as they passed in
> CI it's possibly an issue with my local setup locally.
>
> Running hive-3 test suite alone, the same tests failed multiple times but
> again might be something to do with my computer / JVM configuration.
>
> *I am -1 (non-binding)*, primarily based on the detached commit (as I had
> quite a good bit of trouble trying to fetch it through my normal processes)
> as well as the failing hive3 tests (though that's not exactly within my
> area of expertise).
>
> If the hive3 test failures are only something that occurs for me, then if
> we fix the "Add version.txt commit" in branch 0.3.x such that when I fetch
> branch 0.3.x it's present, I'd be +1. Unfortunately, I can't help with
> cleaning up with the release branch outside of advising somebody else (if
> desired), but I'm happy to help with that.
>
> The hive3 test failures for me seem to be OOM related, but I raised my
>
> Find attached a picture of the detached commit ID,
> *0784d64a659abd4fdaa82cdb599a250a7514facf*, per Github.
>
> [image: image.png]
>
> Example test failures
> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine >
> testCBOWithSelectedColumnsOverlapJoin[fileFormat=AVRO, engine=tez,
> catalog=HIVE_CATALOG, isVectorized=false] FAILED
> java.lang.IllegalArgumentException: Failed to execute Hive query
> 'SELECT c.first_name, o.order_id FROM default.orders o JOIN
> default.customers c ON o.customer_id = c.customer_id ORDER BY o.order_id
> DESC': Error while processing statement: FAILED: Execution Error, return
> code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
> at
> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:152)
> at
> org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine.testCBOWithSelectedColumnsOverlapJoin(TestHiveIcebergStorageHandlerWithEngine.java:236)
>
> Caused by:
> org.apache.hive.service.cli.HiveSQLException: Error while
> processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:263)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:510)
> at
> org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:267)
> at
> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:139)
> ... 1 more
>
> Thanks you for working on this,
> Kyle
>
>
>
>
> On Wed, Jun 1, 2022 at 11:12 PM Eduard Tudenhoefner <ed...@dremio.com>
> wrote:
>
>> Hi Everyone,
>>
>> I propose that we release the following RC as the official Apache Iceberg
>> 0.13.2 release.
>>
>> The commit ID is *0784d64a659abd4fdaa82cdb599a250a7514facf*
>>
>>
>> - This corresponds to the tag: *apache-iceberg-0.13.2-rc1*
>> - https://github.com/apache/iceberg/commits/apache-iceberg-0.13.2-rc1
>> -
>> https://github.com/apache/iceberg/tree/0784d64a659abd4fdaa82cdb599a250a7514facf
>>
>>
>> The release tarball, signature, and checksums are here:
>>
>> -
>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.2-rc1
>>
>>
>> You can find the KEYS file here:
>>
>> - https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>
>>
>> Convenience binary artifacts are staged on Nexus. The Maven repository
>> URL is:
>>
>> -
>> https://repository.apache.org/content/repositories/orgapacheiceberg-1088/
>>
>>
>> Please download, verify, and test.
>>
>> Please vote in the next 72 hours.
>>
>> [ ] +1 Release this as Apache Iceberg <VERSION>
>> [ ] +0
>> [ ] -1 Do not release this because...
>>
>
Re: [VOTE] Release Apache Iceberg 0.13.2 RC1
Posted by Kyle Bendickson <ky...@tabular.io>.
Thanks Eduard!
I have:
- verified the signature
- verified the checksum in the file given as well as of the artifact
- ran all unit tests on Java 11, all passed
- ran all unit tests on Java 8, some hive-3 tests consistently fail (I do
notice they passed on Github - but the tests which fail are consistent
despite giving the JVM more memory and checking for OOM)
- ran a simple smoke test suite of CRUD on namespaces and v1 and v2 tables
with Spark (3.2, 3.1) and Flink (1.13 and 1.14).
- ran some upsert related tests on Flink 1.13 and 1.14 (1.12 is provided a
deprecation notice)
*Problems:*
I did notice that the *given commit ID is considered unattached (and I
wasn't able to check it out).* I am running my tests by using the provided
JAR with engines and then running unit tests locally for the commit just
prior (with commit ID *fae977a9f0a79266a04647b0df2ab540cf0dcff4*).
Not sure if this is a huge issue, but outside of this unattached commit, my
only concern is the `iceberg-hive3` failing tests, but as they passed in CI
it's possibly an issue with my local setup locally.
Running hive-3 test suite alone, the same tests failed multiple times but
again might be something to do with my computer / JVM configuration.
*I am -1 (non-binding)*, primarily based on the detached commit (as I had
quite a good bit of trouble trying to fetch it through my normal processes)
as well as the failing hive3 tests (though that's not exactly within my
area of expertise).
If the hive3 test failures are only something that occurs for me, then if
we fix the "Add version.txt commit" in branch 0.3.x such that when I fetch
branch 0.3.x it's present, I'd be +1. Unfortunately, I can't help with
cleaning up with the release branch outside of advising somebody else (if
desired), but I'm happy to help with that.
The hive3 test failures for me seem to be OOM related, but I raised my
Find attached a picture of the detached commit ID,
*0784d64a659abd4fdaa82cdb599a250a7514facf*, per Github.
[image: image.png]
Example test failures
org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine >
testCBOWithSelectedColumnsOverlapJoin[fileFormat=AVRO, engine=tez,
catalog=HIVE_CATALOG, isVectorized=false] FAILED
java.lang.IllegalArgumentException: Failed to execute Hive query
'SELECT c.first_name, o.order_id FROM default.orders o JOIN
default.customers c ON o.customer_id = c.customer_id ORDER BY o.order_id
DESC': Error while processing statement: FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
at
org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:152)
at
org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine.testCBOWithSelectedColumnsOverlapJoin(TestHiveIcebergStorageHandlerWithEngine.java:236)
Caused by:
org.apache.hive.service.cli.HiveSQLException: Error while
processing statement: FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.tez.TezTask
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:263)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:510)
at
org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:267)
at
org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:139)
... 1 more
Thanks you for working on this,
Kyle
On Wed, Jun 1, 2022 at 11:12 PM Eduard Tudenhoefner <ed...@dremio.com>
wrote:
> Hi Everyone,
>
> I propose that we release the following RC as the official Apache Iceberg
> 0.13.2 release.
>
> The commit ID is *0784d64a659abd4fdaa82cdb599a250a7514facf*
>
>
> - This corresponds to the tag: *apache-iceberg-0.13.2-rc1*
> - https://github.com/apache/iceberg/commits/apache-iceberg-0.13.2-rc1
> -
> https://github.com/apache/iceberg/tree/0784d64a659abd4fdaa82cdb599a250a7514facf
>
>
> The release tarball, signature, and checksums are here:
>
> -
> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.2-rc1
>
>
> You can find the KEYS file here:
>
> - https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>
>
> Convenience binary artifacts are staged on Nexus. The Maven repository URL
> is:
>
> -
> https://repository.apache.org/content/repositories/orgapacheiceberg-1088/
>
>
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as Apache Iceberg <VERSION>
> [ ] +0
> [ ] -1 Do not release this because...
>