You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Ryan Blue <bl...@apache.org> on 2020/04/30 01:29:11 UTC
[VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Hi everyone,
I propose the following candidate to be released as the official Apache
Iceberg 0.8.0-incubating release.
The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
* This corresponds to the tag: apache-iceberg-0.8.0-incubating-rc2
*
https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
* https://github.com/apache/incubator-iceberg/tree/8c05a2f5
The release tarball, signature, and checksums are here:
*
https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
Convenience binary artifacts are staged in Nexus. The Maven repository URL
is:
* https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
This release contains many bug fixes and several new features:
* Actions to remove orphaned files and to optimize metadata for query
performance
* Support for ORC data files
* Snapshot cherry-picking
* Incremental scan planning based on table history
* In and notIn expressions
* An InputFormat for writing MR jobs
Please download, verify, and test.
Please vote in the next 72 hours.
[ ] +1 Release this as Apache Iceberg 0.8.0-incubating
[ ] +0
[ ] -1 Do not release this because...
--
Ryan Blue
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by Jacques Nadeau <ja...@dremio.com>.
+1 (non-binding)
Ran through steps 1-7, completed successfully.
I also updated Nessie to pull from the staging maven repository and ran the
Nessie test suite and it completed successfully with the staged 0.10.0
artifacts.
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Sat, May 2, 2020 at 8:54 PM tison <wa...@gmail.com> wrote:
> +1 (non-binding)
>
> √ RAT checks passed
> √ signature is correct
> √ checksum is correct
> √ build from source (with java 8)
> √ run tests locally
>
> Best,
> tison.
>
>
> Carl Steinbach <cw...@apache.org> 于2020年5月3日周日 上午11:09写道：
>
>> +1 (binding)
>>
>>
>> On Fri, May 1, 2020 at 9:38 AM RD <rd...@gmail.com> wrote:
>>
>>> +1
>>> Validated all the steps mentioned.
>>>
>>> -R
>>>
>>> On Fri, May 1, 2020 at 9:31 AM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>>> +1 (binding)
>>>>
>>>> Ran rat, validated checksums and signature, and ran the build.
>>>>
>>>> I noticed that the iceberg-spark-runtime Jar is about 22MB larger and
>>>> it looks like the problem is mainly that parquet-avro 1.11.0 is shading all
>>>> of fastutil without minimizing the Jar like parquet-column does. I tried
>>>> rolling back to 1.10.1, but that requires rolling back Avro as well, so I
>>>> think the best option right now is to continue with a 37MB runtime Jar. We
>>>> can fix this in a 0.8.1 release when Parquet releases 1.11.1 with a fix.
>>>>
>>>> rb
>>>>
>>>> On Thu, Apr 30, 2020 at 11:41 PM Gautam <ga...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> Ran checks on
>>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>>>
>>>>> √ RAT checks passed
>>>>> √ signature is correct
>>>>> √ checksum is correct
>>>>> √ build from source (with java 8)
>>>>> √ run tests locally
>>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Apr 30, 2020 at 4:18 PM Samarth Jain <sa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> +1 (non-binding)
>>>>>> all checks passed
>>>>>>
>>>>>> On Thu, Apr 30, 2020 at 4:06 PM John Zhuge <jz...@apache.org> wrote:
>>>>>>
>>>>>>> +1 (non-binding)
>>>>>>>
>>>>>>> 1. Checked signature and checksum
>>>>>>> 2. Checked license
>>>>>>> 3. Built and ran unit tests.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Apr 30, 2020 at 2:24 PM Owen O'Malley <
>>>>>>> owen.omalley@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>> 1. Checked signature and checksum
>>>>>>>> 2. Built and ran unit tests.
>>>>>>>> 3. Checked ORC version :)
>>>>>>>>
>>>>>>>> On Monday, ORC released 1.6.3, so we should grab those fixes soon.
>>>>>>>>
>>>>>>>> .. Owen
>>>>>>>>
>>>>>>>> On Thu, Apr 30, 2020 at 12:34 PM Dongjoon Hyun <
>>>>>>>> dongjoon.hyun@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> +1.
>>>>>>>>>
>>>>>>>>> 1. Verified checksum, sig, and license
>>>>>>>>> 3. Build from the source and run UTs.
>>>>>>>>> 4. Run some manual ORC write/read tests with Apache Spark
>>>>>>>>> 2.4.6-SNAPSHOT (as of today).
>>>>>>>>>
>>>>>>>>> Thank you, all!
>>>>>>>>>
>>>>>>>>> Bests,
>>>>>>>>> Dongjoon.
>>>>>>>>>
>>>>>>>>> On Thu, Apr 30, 2020 at 10:28 AM parth brahmbhatt <
>>>>>>>>> brahmbhatt.parth@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> +1. checks passed, did not observe the unit test failure.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Parth
>>>>>>>>>>
>>>>>>>>>> On Thu, Apr 30, 2020 at 9:13 AM Daniel Weeks <dw...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> +1 all checks passed
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Apr 30, 2020 at 8:53 AM Anton Okolnychyi
>>>>>>>>>>> <ao...@apple.com.invalid> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> That test uses many concurrent writes and I’ve seen cases when
>>>>>>>>>>>> it led to deadlocks in our test HMS. I think HMS is capable of recovering
>>>>>>>>>>>> on its own but that process can be slow in highly concurrent environments.
>>>>>>>>>>>> There is a 2 min timeout in that test so it can potentially fail. I’ve seen
>>>>>>>>>>>> a deadlock but 2 min was always enough for that test in my local env and
>>>>>>>>>>>> internal/upstream build pipelines. If there is an environment that
>>>>>>>>>>>> constantly or frequently hits this problem, it would be great to check
>>>>>>>>>>>> debug logs.
>>>>>>>>>>>>
>>>>>>>>>>>> I am +1 on releasing RC2. I checked it locally.
>>>>>>>>>>>>
>>>>>>>>>>>> - Anton
>>>>>>>>>>>>
>>>>>>>>>>>> On 30 Apr 2020, at 02:52, Mass Dosage <ma...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> The build for RC2 worked fine for me, I didn't get a failure on
>>>>>>>>>>>> "TestHiveTableConcurrency". Perhaps there is some kind of race condition in
>>>>>>>>>>>> the test? I have seen timeout errors like that when I ran tests on an
>>>>>>>>>>>> overloaded machine, could that have been the case?
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, 30 Apr 2020 at 08:32, OpenInx <op...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I checked the rc2, seems the TestHiveTableConcurrency is
>>>>>>>>>>>>> broken, may need to fix it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. Download the tarball and check the signature & checksum: OK
>>>>>>>>>>>>> 2. license checking: RAT checks passed.
>>>>>>>>>>>>> 3. Build and test the project (java8):
>>>>>>>>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency >
>>>>>>>>>>>>> testConcurrentConnections FAILED
>>>>>>>>>>>>> java.lang.AssertionError: Timeout
>>>>>>>>>>>>> at org.junit.Assert.fail(Assert.java:88)
>>>>>>>>>>>>> at org.junit.Assert.assertTrue(Assert.java:41)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <bl...@apache.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I propose the following candidate to be released as the
>>>>>>>>>>>>>> official Apache Iceberg 0.8.0-incubating release.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
>>>>>>>>>>>>>> * This corresponds to the tag:
>>>>>>>>>>>>>> apache-iceberg-0.8.0-incubating-rc2
>>>>>>>>>>>>>> *
>>>>>>>>>>>>>> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
>>>>>>>>>>>>>> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The release tarball, signature, and checksums are here:
>>>>>>>>>>>>>> *
>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can find the KEYS file here:
>>>>>>>>>>>>>> *
>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Convenience binary artifacts are staged in Nexus. The Maven
>>>>>>>>>>>>>> repository URL is:
>>>>>>>>>>>>>> *
>>>>>>>>>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This release contains many bug fixes and several new features:
>>>>>>>>>>>>>> * Actions to remove orphaned files and to optimize metadata
>>>>>>>>>>>>>> for query performance
>>>>>>>>>>>>>> * Support for ORC data files
>>>>>>>>>>>>>> * Snapshot cherry-picking
>>>>>>>>>>>>>> * Incremental scan planning based on table history
>>>>>>>>>>>>>> * In and notIn expressions
>>>>>>>>>>>>>> * An InputFormat for writing MR jobs
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please download, verify, and test.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please vote in the next 72 hours.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
>>>>>>>>>>>>>> [ ] +0
>>>>>>>>>>>>>> [ ] -1 Do not release this because...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> John Zhuge
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by tison <wa...@gmail.com>.
+1 (non-binding)
√ RAT checks passed
√ signature is correct
√ checksum is correct
√ build from source (with java 8)
√ run tests locally
Best,
tison.
Carl Steinbach <cw...@apache.org> 于2020年5月3日周日 上午11:09写道：
> +1 (binding)
>
>
> On Fri, May 1, 2020 at 9:38 AM RD <rd...@gmail.com> wrote:
>
>> +1
>> Validated all the steps mentioned.
>>
>> -R
>>
>> On Fri, May 1, 2020 at 9:31 AM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>>> +1 (binding)
>>>
>>> Ran rat, validated checksums and signature, and ran the build.
>>>
>>> I noticed that the iceberg-spark-runtime Jar is about 22MB larger and it
>>> looks like the problem is mainly that parquet-avro 1.11.0 is shading all of
>>> fastutil without minimizing the Jar like parquet-column does. I tried
>>> rolling back to 1.10.1, but that requires rolling back Avro as well, so I
>>> think the best option right now is to continue with a 37MB runtime Jar. We
>>> can fix this in a 0.8.1 release when Parquet releases 1.11.1 with a fix.
>>>
>>> rb
>>>
>>> On Thu, Apr 30, 2020 at 11:41 PM Gautam <ga...@gmail.com> wrote:
>>>
>>>>
>>>> Ran checks on
>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>>
>>>> √ RAT checks passed
>>>> √ signature is correct
>>>> √ checksum is correct
>>>> √ build from source (with java 8)
>>>> √ run tests locally
>>>>
>>>> +1 (non-binding)
>>>>
>>>>
>>>>
>>>> On Thu, Apr 30, 2020 at 4:18 PM Samarth Jain <sa...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1 (non-binding)
>>>>> all checks passed
>>>>>
>>>>> On Thu, Apr 30, 2020 at 4:06 PM John Zhuge <jz...@apache.org> wrote:
>>>>>
>>>>>> +1 (non-binding)
>>>>>>
>>>>>> 1. Checked signature and checksum
>>>>>> 2. Checked license
>>>>>> 3. Built and ran unit tests.
>>>>>>
>>>>>>
>>>>>> On Thu, Apr 30, 2020 at 2:24 PM Owen O'Malley <ow...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> 1. Checked signature and checksum
>>>>>>> 2. Built and ran unit tests.
>>>>>>> 3. Checked ORC version :)
>>>>>>>
>>>>>>> On Monday, ORC released 1.6.3, so we should grab those fixes soon.
>>>>>>>
>>>>>>> .. Owen
>>>>>>>
>>>>>>> On Thu, Apr 30, 2020 at 12:34 PM Dongjoon Hyun <
>>>>>>> dongjoon.hyun@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1.
>>>>>>>>
>>>>>>>> 1. Verified checksum, sig, and license
>>>>>>>> 3. Build from the source and run UTs.
>>>>>>>> 4. Run some manual ORC write/read tests with Apache Spark
>>>>>>>> 2.4.6-SNAPSHOT (as of today).
>>>>>>>>
>>>>>>>> Thank you, all!
>>>>>>>>
>>>>>>>> Bests,
>>>>>>>> Dongjoon.
>>>>>>>>
>>>>>>>> On Thu, Apr 30, 2020 at 10:28 AM parth brahmbhatt <
>>>>>>>> brahmbhatt.parth@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> +1. checks passed, did not observe the unit test failure.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Parth
>>>>>>>>>
>>>>>>>>> On Thu, Apr 30, 2020 at 9:13 AM Daniel Weeks <dw...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> +1 all checks passed
>>>>>>>>>>
>>>>>>>>>> On Thu, Apr 30, 2020 at 8:53 AM Anton Okolnychyi
>>>>>>>>>> <ao...@apple.com.invalid> wrote:
>>>>>>>>>>
>>>>>>>>>>> That test uses many concurrent writes and I’ve seen cases when
>>>>>>>>>>> it led to deadlocks in our test HMS. I think HMS is capable of recovering
>>>>>>>>>>> on its own but that process can be slow in highly concurrent environments.
>>>>>>>>>>> There is a 2 min timeout in that test so it can potentially fail. I’ve seen
>>>>>>>>>>> a deadlock but 2 min was always enough for that test in my local env and
>>>>>>>>>>> internal/upstream build pipelines. If there is an environment that
>>>>>>>>>>> constantly or frequently hits this problem, it would be great to check
>>>>>>>>>>> debug logs.
>>>>>>>>>>>
>>>>>>>>>>> I am +1 on releasing RC2. I checked it locally.
>>>>>>>>>>>
>>>>>>>>>>> - Anton
>>>>>>>>>>>
>>>>>>>>>>> On 30 Apr 2020, at 02:52, Mass Dosage <ma...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> The build for RC2 worked fine for me, I didn't get a failure on
>>>>>>>>>>> "TestHiveTableConcurrency". Perhaps there is some kind of race condition in
>>>>>>>>>>> the test? I have seen timeout errors like that when I ran tests on an
>>>>>>>>>>> overloaded machine, could that have been the case?
>>>>>>>>>>>
>>>>>>>>>>> On Thu, 30 Apr 2020 at 08:32, OpenInx <op...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I checked the rc2, seems the TestHiveTableConcurrency is
>>>>>>>>>>>> broken, may need to fix it.
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Download the tarball and check the signature & checksum: OK
>>>>>>>>>>>> 2. license checking: RAT checks passed.
>>>>>>>>>>>> 3. Build and test the project (java8):
>>>>>>>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency >
>>>>>>>>>>>> testConcurrentConnections FAILED
>>>>>>>>>>>> java.lang.AssertionError: Timeout
>>>>>>>>>>>> at org.junit.Assert.fail(Assert.java:88)
>>>>>>>>>>>> at org.junit.Assert.assertTrue(Assert.java:41)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <bl...@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I propose the following candidate to be released as the
>>>>>>>>>>>>> official Apache Iceberg 0.8.0-incubating release.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
>>>>>>>>>>>>> * This corresponds to the tag:
>>>>>>>>>>>>> apache-iceberg-0.8.0-incubating-rc2
>>>>>>>>>>>>> *
>>>>>>>>>>>>> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
>>>>>>>>>>>>> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5
>>>>>>>>>>>>>
>>>>>>>>>>>>> The release tarball, signature, and checksums are here:
>>>>>>>>>>>>> *
>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can find the KEYS file here:
>>>>>>>>>>>>> *
>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
>>>>>>>>>>>>>
>>>>>>>>>>>>> Convenience binary artifacts are staged in Nexus. The Maven
>>>>>>>>>>>>> repository URL is:
>>>>>>>>>>>>> *
>>>>>>>>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
>>>>>>>>>>>>>
>>>>>>>>>>>>> This release contains many bug fixes and several new features:
>>>>>>>>>>>>> * Actions to remove orphaned files and to optimize metadata
>>>>>>>>>>>>> for query performance
>>>>>>>>>>>>> * Support for ORC data files
>>>>>>>>>>>>> * Snapshot cherry-picking
>>>>>>>>>>>>> * Incremental scan planning based on table history
>>>>>>>>>>>>> * In and notIn expressions
>>>>>>>>>>>>> * An InputFormat for writing MR jobs
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please download, verify, and test.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please vote in the next 72 hours.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
>>>>>>>>>>>>> [ ] +0
>>>>>>>>>>>>> [ ] -1 Do not release this because...
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> John Zhuge
>>>>>>
>>>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by Carl Steinbach <cw...@apache.org>.
+1 (binding)
On Fri, May 1, 2020 at 9:38 AM RD <rd...@gmail.com> wrote:
> +1
> Validated all the steps mentioned.
>
> -R
>
> On Fri, May 1, 2020 at 9:31 AM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> +1 (binding)
>>
>> Ran rat, validated checksums and signature, and ran the build.
>>
>> I noticed that the iceberg-spark-runtime Jar is about 22MB larger and it
>> looks like the problem is mainly that parquet-avro 1.11.0 is shading all of
>> fastutil without minimizing the Jar like parquet-column does. I tried
>> rolling back to 1.10.1, but that requires rolling back Avro as well, so I
>> think the best option right now is to continue with a 37MB runtime Jar. We
>> can fix this in a 0.8.1 release when Parquet releases 1.11.1 with a fix.
>>
>> rb
>>
>> On Thu, Apr 30, 2020 at 11:41 PM Gautam <ga...@gmail.com> wrote:
>>
>>>
>>> Ran checks on
>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>
>>> √ RAT checks passed
>>> √ signature is correct
>>> √ checksum is correct
>>> √ build from source (with java 8)
>>> √ run tests locally
>>>
>>> +1 (non-binding)
>>>
>>>
>>>
>>> On Thu, Apr 30, 2020 at 4:18 PM Samarth Jain <sa...@gmail.com>
>>> wrote:
>>>
>>>> +1 (non-binding)
>>>> all checks passed
>>>>
>>>> On Thu, Apr 30, 2020 at 4:06 PM John Zhuge <jz...@apache.org> wrote:
>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>> 1. Checked signature and checksum
>>>>> 2. Checked license
>>>>> 3. Built and ran unit tests.
>>>>>
>>>>>
>>>>> On Thu, Apr 30, 2020 at 2:24 PM Owen O'Malley <ow...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> 1. Checked signature and checksum
>>>>>> 2. Built and ran unit tests.
>>>>>> 3. Checked ORC version :)
>>>>>>
>>>>>> On Monday, ORC released 1.6.3, so we should grab those fixes soon.
>>>>>>
>>>>>> .. Owen
>>>>>>
>>>>>> On Thu, Apr 30, 2020 at 12:34 PM Dongjoon Hyun <
>>>>>> dongjoon.hyun@gmail.com> wrote:
>>>>>>
>>>>>>> +1.
>>>>>>>
>>>>>>> 1. Verified checksum, sig, and license
>>>>>>> 3. Build from the source and run UTs.
>>>>>>> 4. Run some manual ORC write/read tests with Apache Spark
>>>>>>> 2.4.6-SNAPSHOT (as of today).
>>>>>>>
>>>>>>> Thank you, all!
>>>>>>>
>>>>>>> Bests,
>>>>>>> Dongjoon.
>>>>>>>
>>>>>>> On Thu, Apr 30, 2020 at 10:28 AM parth brahmbhatt <
>>>>>>> brahmbhatt.parth@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1. checks passed, did not observe the unit test failure.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Parth
>>>>>>>>
>>>>>>>> On Thu, Apr 30, 2020 at 9:13 AM Daniel Weeks <dw...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1 all checks passed
>>>>>>>>>
>>>>>>>>> On Thu, Apr 30, 2020 at 8:53 AM Anton Okolnychyi
>>>>>>>>> <ao...@apple.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> That test uses many concurrent writes and I’ve seen cases when it
>>>>>>>>>> led to deadlocks in our test HMS. I think HMS is capable of recovering on
>>>>>>>>>> its own but that process can be slow in highly concurrent environments.
>>>>>>>>>> There is a 2 min timeout in that test so it can potentially fail. I’ve seen
>>>>>>>>>> a deadlock but 2 min was always enough for that test in my local env and
>>>>>>>>>> internal/upstream build pipelines. If there is an environment that
>>>>>>>>>> constantly or frequently hits this problem, it would be great to check
>>>>>>>>>> debug logs.
>>>>>>>>>>
>>>>>>>>>> I am +1 on releasing RC2. I checked it locally.
>>>>>>>>>>
>>>>>>>>>> - Anton
>>>>>>>>>>
>>>>>>>>>> On 30 Apr 2020, at 02:52, Mass Dosage <ma...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> The build for RC2 worked fine for me, I didn't get a failure on
>>>>>>>>>> "TestHiveTableConcurrency". Perhaps there is some kind of race condition in
>>>>>>>>>> the test? I have seen timeout errors like that when I ran tests on an
>>>>>>>>>> overloaded machine, could that have been the case?
>>>>>>>>>>
>>>>>>>>>> On Thu, 30 Apr 2020 at 08:32, OpenInx <op...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I checked the rc2, seems the TestHiveTableConcurrency is broken,
>>>>>>>>>>> may need to fix it.
>>>>>>>>>>>
>>>>>>>>>>> 1. Download the tarball and check the signature & checksum: OK
>>>>>>>>>>> 2. license checking: RAT checks passed.
>>>>>>>>>>> 3. Build and test the project (java8):
>>>>>>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency >
>>>>>>>>>>> testConcurrentConnections FAILED
>>>>>>>>>>> java.lang.AssertionError: Timeout
>>>>>>>>>>> at org.junit.Assert.fail(Assert.java:88)
>>>>>>>>>>> at org.junit.Assert.assertTrue(Assert.java:41)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <bl...@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>
>>>>>>>>>>>> I propose the following candidate to be released as the
>>>>>>>>>>>> official Apache Iceberg 0.8.0-incubating release.
>>>>>>>>>>>>
>>>>>>>>>>>> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
>>>>>>>>>>>> * This corresponds to the tag:
>>>>>>>>>>>> apache-iceberg-0.8.0-incubating-rc2
>>>>>>>>>>>> *
>>>>>>>>>>>> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
>>>>>>>>>>>> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5
>>>>>>>>>>>>
>>>>>>>>>>>> The release tarball, signature, and checksums are here:
>>>>>>>>>>>> *
>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>>>>>>>>>>
>>>>>>>>>>>> You can find the KEYS file here:
>>>>>>>>>>>> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
>>>>>>>>>>>>
>>>>>>>>>>>> Convenience binary artifacts are staged in Nexus. The Maven
>>>>>>>>>>>> repository URL is:
>>>>>>>>>>>> *
>>>>>>>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
>>>>>>>>>>>>
>>>>>>>>>>>> This release contains many bug fixes and several new features:
>>>>>>>>>>>> * Actions to remove orphaned files and to optimize metadata for
>>>>>>>>>>>> query performance
>>>>>>>>>>>> * Support for ORC data files
>>>>>>>>>>>> * Snapshot cherry-picking
>>>>>>>>>>>> * Incremental scan planning based on table history
>>>>>>>>>>>> * In and notIn expressions
>>>>>>>>>>>> * An InputFormat for writing MR jobs
>>>>>>>>>>>>
>>>>>>>>>>>> Please download, verify, and test.
>>>>>>>>>>>>
>>>>>>>>>>>> Please vote in the next 72 hours.
>>>>>>>>>>>>
>>>>>>>>>>>> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
>>>>>>>>>>>> [ ] +0
>>>>>>>>>>>> [ ] -1 Do not release this because...
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>
>>>>> --
>>>>> John Zhuge
>>>>>
>>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by RD <rd...@gmail.com>.
+1
Validated all the steps mentioned.
-R
On Fri, May 1, 2020 at 9:31 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
> +1 (binding)
>
> Ran rat, validated checksums and signature, and ran the build.
>
> I noticed that the iceberg-spark-runtime Jar is about 22MB larger and it
> looks like the problem is mainly that parquet-avro 1.11.0 is shading all of
> fastutil without minimizing the Jar like parquet-column does. I tried
> rolling back to 1.10.1, but that requires rolling back Avro as well, so I
> think the best option right now is to continue with a 37MB runtime Jar. We
> can fix this in a 0.8.1 release when Parquet releases 1.11.1 with a fix.
>
> rb
>
> On Thu, Apr 30, 2020 at 11:41 PM Gautam <ga...@gmail.com> wrote:
>
>>
>> Ran checks on
>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>
>> √ RAT checks passed
>> √ signature is correct
>> √ checksum is correct
>> √ build from source (with java 8)
>> √ run tests locally
>>
>> +1 (non-binding)
>>
>>
>>
>> On Thu, Apr 30, 2020 at 4:18 PM Samarth Jain <sa...@gmail.com>
>> wrote:
>>
>>> +1 (non-binding)
>>> all checks passed
>>>
>>> On Thu, Apr 30, 2020 at 4:06 PM John Zhuge <jz...@apache.org> wrote:
>>>
>>>> +1 (non-binding)
>>>>
>>>> 1. Checked signature and checksum
>>>> 2. Checked license
>>>> 3. Built and ran unit tests.
>>>>
>>>>
>>>> On Thu, Apr 30, 2020 at 2:24 PM Owen O'Malley <ow...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> 1. Checked signature and checksum
>>>>> 2. Built and ran unit tests.
>>>>> 3. Checked ORC version :)
>>>>>
>>>>> On Monday, ORC released 1.6.3, so we should grab those fixes soon.
>>>>>
>>>>> .. Owen
>>>>>
>>>>> On Thu, Apr 30, 2020 at 12:34 PM Dongjoon Hyun <
>>>>> dongjoon.hyun@gmail.com> wrote:
>>>>>
>>>>>> +1.
>>>>>>
>>>>>> 1. Verified checksum, sig, and license
>>>>>> 3. Build from the source and run UTs.
>>>>>> 4. Run some manual ORC write/read tests with Apache Spark
>>>>>> 2.4.6-SNAPSHOT (as of today).
>>>>>>
>>>>>> Thank you, all!
>>>>>>
>>>>>> Bests,
>>>>>> Dongjoon.
>>>>>>
>>>>>> On Thu, Apr 30, 2020 at 10:28 AM parth brahmbhatt <
>>>>>> brahmbhatt.parth@gmail.com> wrote:
>>>>>>
>>>>>>> +1. checks passed, did not observe the unit test failure.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Parth
>>>>>>>
>>>>>>> On Thu, Apr 30, 2020 at 9:13 AM Daniel Weeks <dw...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1 all checks passed
>>>>>>>>
>>>>>>>> On Thu, Apr 30, 2020 at 8:53 AM Anton Okolnychyi
>>>>>>>> <ao...@apple.com.invalid> wrote:
>>>>>>>>
>>>>>>>>> That test uses many concurrent writes and I’ve seen cases when it
>>>>>>>>> led to deadlocks in our test HMS. I think HMS is capable of recovering on
>>>>>>>>> its own but that process can be slow in highly concurrent environments.
>>>>>>>>> There is a 2 min timeout in that test so it can potentially fail. I’ve seen
>>>>>>>>> a deadlock but 2 min was always enough for that test in my local env and
>>>>>>>>> internal/upstream build pipelines. If there is an environment that
>>>>>>>>> constantly or frequently hits this problem, it would be great to check
>>>>>>>>> debug logs.
>>>>>>>>>
>>>>>>>>> I am +1 on releasing RC2. I checked it locally.
>>>>>>>>>
>>>>>>>>> - Anton
>>>>>>>>>
>>>>>>>>> On 30 Apr 2020, at 02:52, Mass Dosage <ma...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> The build for RC2 worked fine for me, I didn't get a failure on
>>>>>>>>> "TestHiveTableConcurrency". Perhaps there is some kind of race condition in
>>>>>>>>> the test? I have seen timeout errors like that when I ran tests on an
>>>>>>>>> overloaded machine, could that have been the case?
>>>>>>>>>
>>>>>>>>> On Thu, 30 Apr 2020 at 08:32, OpenInx <op...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I checked the rc2, seems the TestHiveTableConcurrency is broken,
>>>>>>>>>> may need to fix it.
>>>>>>>>>>
>>>>>>>>>> 1. Download the tarball and check the signature & checksum: OK
>>>>>>>>>> 2. license checking: RAT checks passed.
>>>>>>>>>> 3. Build and test the project (java8):
>>>>>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency >
>>>>>>>>>> testConcurrentConnections FAILED
>>>>>>>>>> java.lang.AssertionError: Timeout
>>>>>>>>>> at org.junit.Assert.fail(Assert.java:88)
>>>>>>>>>> at org.junit.Assert.assertTrue(Assert.java:41)
>>>>>>>>>> at
>>>>>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
>>>>>>>>>>
>>>>>>>>>> On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <bl...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>
>>>>>>>>>>> I propose the following candidate to be released as the official
>>>>>>>>>>> Apache Iceberg 0.8.0-incubating release.
>>>>>>>>>>>
>>>>>>>>>>> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
>>>>>>>>>>> * This corresponds to the tag:
>>>>>>>>>>> apache-iceberg-0.8.0-incubating-rc2
>>>>>>>>>>> *
>>>>>>>>>>> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
>>>>>>>>>>> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5
>>>>>>>>>>>
>>>>>>>>>>> The release tarball, signature, and checksums are here:
>>>>>>>>>>> *
>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>>>>>>>>>
>>>>>>>>>>> You can find the KEYS file here:
>>>>>>>>>>> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
>>>>>>>>>>>
>>>>>>>>>>> Convenience binary artifacts are staged in Nexus. The Maven
>>>>>>>>>>> repository URL is:
>>>>>>>>>>> *
>>>>>>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
>>>>>>>>>>>
>>>>>>>>>>> This release contains many bug fixes and several new features:
>>>>>>>>>>> * Actions to remove orphaned files and to optimize metadata for
>>>>>>>>>>> query performance
>>>>>>>>>>> * Support for ORC data files
>>>>>>>>>>> * Snapshot cherry-picking
>>>>>>>>>>> * Incremental scan planning based on table history
>>>>>>>>>>> * In and notIn expressions
>>>>>>>>>>> * An InputFormat for writing MR jobs
>>>>>>>>>>>
>>>>>>>>>>> Please download, verify, and test.
>>>>>>>>>>>
>>>>>>>>>>> Please vote in the next 72 hours.
>>>>>>>>>>>
>>>>>>>>>>> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
>>>>>>>>>>> [ ] +0
>>>>>>>>>>> [ ] -1 Do not release this because...
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>
>>>> --
>>>> John Zhuge
>>>>
>>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
+1 (binding)
Ran rat, validated checksums and signature, and ran the build.
I noticed that the iceberg-spark-runtime Jar is about 22MB larger and it
looks like the problem is mainly that parquet-avro 1.11.0 is shading all of
fastutil without minimizing the Jar like parquet-column does. I tried
rolling back to 1.10.1, but that requires rolling back Avro as well, so I
think the best option right now is to continue with a 37MB runtime Jar. We
can fix this in a 0.8.1 release when Parquet releases 1.11.1 with a fix.
rb
On Thu, Apr 30, 2020 at 11:41 PM Gautam <ga...@gmail.com> wrote:
>
> Ran checks on
> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>
> √ RAT checks passed
> √ signature is correct
> √ checksum is correct
> √ build from source (with java 8)
> √ run tests locally
>
> +1 (non-binding)
>
>
>
> On Thu, Apr 30, 2020 at 4:18 PM Samarth Jain <sa...@gmail.com>
> wrote:
>
>> +1 (non-binding)
>> all checks passed
>>
>> On Thu, Apr 30, 2020 at 4:06 PM John Zhuge <jz...@apache.org> wrote:
>>
>>> +1 (non-binding)
>>>
>>> 1. Checked signature and checksum
>>> 2. Checked license
>>> 3. Built and ran unit tests.
>>>
>>>
>>> On Thu, Apr 30, 2020 at 2:24 PM Owen O'Malley <ow...@gmail.com>
>>> wrote:
>>>
>>>> +1
>>>>
>>>> 1. Checked signature and checksum
>>>> 2. Built and ran unit tests.
>>>> 3. Checked ORC version :)
>>>>
>>>> On Monday, ORC released 1.6.3, so we should grab those fixes soon.
>>>>
>>>> .. Owen
>>>>
>>>> On Thu, Apr 30, 2020 at 12:34 PM Dongjoon Hyun <do...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1.
>>>>>
>>>>> 1. Verified checksum, sig, and license
>>>>> 3. Build from the source and run UTs.
>>>>> 4. Run some manual ORC write/read tests with Apache Spark
>>>>> 2.4.6-SNAPSHOT (as of today).
>>>>>
>>>>> Thank you, all!
>>>>>
>>>>> Bests,
>>>>> Dongjoon.
>>>>>
>>>>> On Thu, Apr 30, 2020 at 10:28 AM parth brahmbhatt <
>>>>> brahmbhatt.parth@gmail.com> wrote:
>>>>>
>>>>>> +1. checks passed, did not observe the unit test failure.
>>>>>>
>>>>>> Thanks
>>>>>> Parth
>>>>>>
>>>>>> On Thu, Apr 30, 2020 at 9:13 AM Daniel Weeks <dw...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> +1 all checks passed
>>>>>>>
>>>>>>> On Thu, Apr 30, 2020 at 8:53 AM Anton Okolnychyi
>>>>>>> <ao...@apple.com.invalid> wrote:
>>>>>>>
>>>>>>>> That test uses many concurrent writes and I’ve seen cases when it
>>>>>>>> led to deadlocks in our test HMS. I think HMS is capable of recovering on
>>>>>>>> its own but that process can be slow in highly concurrent environments.
>>>>>>>> There is a 2 min timeout in that test so it can potentially fail. I’ve seen
>>>>>>>> a deadlock but 2 min was always enough for that test in my local env and
>>>>>>>> internal/upstream build pipelines. If there is an environment that
>>>>>>>> constantly or frequently hits this problem, it would be great to check
>>>>>>>> debug logs.
>>>>>>>>
>>>>>>>> I am +1 on releasing RC2. I checked it locally.
>>>>>>>>
>>>>>>>> - Anton
>>>>>>>>
>>>>>>>> On 30 Apr 2020, at 02:52, Mass Dosage <ma...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> The build for RC2 worked fine for me, I didn't get a failure on
>>>>>>>> "TestHiveTableConcurrency". Perhaps there is some kind of race condition in
>>>>>>>> the test? I have seen timeout errors like that when I ran tests on an
>>>>>>>> overloaded machine, could that have been the case?
>>>>>>>>
>>>>>>>> On Thu, 30 Apr 2020 at 08:32, OpenInx <op...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I checked the rc2, seems the TestHiveTableConcurrency is broken,
>>>>>>>>> may need to fix it.
>>>>>>>>>
>>>>>>>>> 1. Download the tarball and check the signature & checksum: OK
>>>>>>>>> 2. license checking: RAT checks passed.
>>>>>>>>> 3. Build and test the project (java8):
>>>>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency >
>>>>>>>>> testConcurrentConnections FAILED
>>>>>>>>> java.lang.AssertionError: Timeout
>>>>>>>>> at org.junit.Assert.fail(Assert.java:88)
>>>>>>>>> at org.junit.Assert.assertTrue(Assert.java:41)
>>>>>>>>> at
>>>>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
>>>>>>>>>
>>>>>>>>> On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <bl...@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> Hi everyone,
>>>>>>>>>>
>>>>>>>>>> I propose the following candidate to be released as the official
>>>>>>>>>> Apache Iceberg 0.8.0-incubating release.
>>>>>>>>>>
>>>>>>>>>> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
>>>>>>>>>> * This corresponds to the tag: apache-iceberg-0.8.0-incubating-rc2
>>>>>>>>>> *
>>>>>>>>>> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
>>>>>>>>>> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5
>>>>>>>>>>
>>>>>>>>>> The release tarball, signature, and checksums are here:
>>>>>>>>>> *
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>>>>>>>>
>>>>>>>>>> You can find the KEYS file here:
>>>>>>>>>> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
>>>>>>>>>>
>>>>>>>>>> Convenience binary artifacts are staged in Nexus. The Maven
>>>>>>>>>> repository URL is:
>>>>>>>>>> *
>>>>>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
>>>>>>>>>>
>>>>>>>>>> This release contains many bug fixes and several new features:
>>>>>>>>>> * Actions to remove orphaned files and to optimize metadata for
>>>>>>>>>> query performance
>>>>>>>>>> * Support for ORC data files
>>>>>>>>>> * Snapshot cherry-picking
>>>>>>>>>> * Incremental scan planning based on table history
>>>>>>>>>> * In and notIn expressions
>>>>>>>>>> * An InputFormat for writing MR jobs
>>>>>>>>>>
>>>>>>>>>> Please download, verify, and test.
>>>>>>>>>>
>>>>>>>>>> Please vote in the next 72 hours.
>>>>>>>>>>
>>>>>>>>>> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
>>>>>>>>>> [ ] +0
>>>>>>>>>> [ ] -1 Do not release this because...
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Ryan Blue
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>
>>> --
>>> John Zhuge
>>>
>>
--
Ryan Blue
Software Engineer
Netflix
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by Gautam <ga...@gmail.com>.
Ran checks on
https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
√ RAT checks passed
√ signature is correct
√ checksum is correct
√ build from source (with java 8)
√ run tests locally
+1 (non-binding)
On Thu, Apr 30, 2020 at 4:18 PM Samarth Jain <sa...@gmail.com> wrote:
> +1 (non-binding)
> all checks passed
>
> On Thu, Apr 30, 2020 at 4:06 PM John Zhuge <jz...@apache.org> wrote:
>
>> +1 (non-binding)
>>
>> 1. Checked signature and checksum
>> 2. Checked license
>> 3. Built and ran unit tests.
>>
>>
>> On Thu, Apr 30, 2020 at 2:24 PM Owen O'Malley <ow...@gmail.com>
>> wrote:
>>
>>> +1
>>>
>>> 1. Checked signature and checksum
>>> 2. Built and ran unit tests.
>>> 3. Checked ORC version :)
>>>
>>> On Monday, ORC released 1.6.3, so we should grab those fixes soon.
>>>
>>> .. Owen
>>>
>>> On Thu, Apr 30, 2020 at 12:34 PM Dongjoon Hyun <do...@gmail.com>
>>> wrote:
>>>
>>>> +1.
>>>>
>>>> 1. Verified checksum, sig, and license
>>>> 3. Build from the source and run UTs.
>>>> 4. Run some manual ORC write/read tests with Apache Spark
>>>> 2.4.6-SNAPSHOT (as of today).
>>>>
>>>> Thank you, all!
>>>>
>>>> Bests,
>>>> Dongjoon.
>>>>
>>>> On Thu, Apr 30, 2020 at 10:28 AM parth brahmbhatt <
>>>> brahmbhatt.parth@gmail.com> wrote:
>>>>
>>>>> +1. checks passed, did not observe the unit test failure.
>>>>>
>>>>> Thanks
>>>>> Parth
>>>>>
>>>>> On Thu, Apr 30, 2020 at 9:13 AM Daniel Weeks <dw...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> +1 all checks passed
>>>>>>
>>>>>> On Thu, Apr 30, 2020 at 8:53 AM Anton Okolnychyi
>>>>>> <ao...@apple.com.invalid> wrote:
>>>>>>
>>>>>>> That test uses many concurrent writes and I’ve seen cases when it
>>>>>>> led to deadlocks in our test HMS. I think HMS is capable of recovering on
>>>>>>> its own but that process can be slow in highly concurrent environments.
>>>>>>> There is a 2 min timeout in that test so it can potentially fail. I’ve seen
>>>>>>> a deadlock but 2 min was always enough for that test in my local env and
>>>>>>> internal/upstream build pipelines. If there is an environment that
>>>>>>> constantly or frequently hits this problem, it would be great to check
>>>>>>> debug logs.
>>>>>>>
>>>>>>> I am +1 on releasing RC2. I checked it locally.
>>>>>>>
>>>>>>> - Anton
>>>>>>>
>>>>>>> On 30 Apr 2020, at 02:52, Mass Dosage <ma...@gmail.com> wrote:
>>>>>>>
>>>>>>> The build for RC2 worked fine for me, I didn't get a failure on
>>>>>>> "TestHiveTableConcurrency". Perhaps there is some kind of race condition in
>>>>>>> the test? I have seen timeout errors like that when I ran tests on an
>>>>>>> overloaded machine, could that have been the case?
>>>>>>>
>>>>>>> On Thu, 30 Apr 2020 at 08:32, OpenInx <op...@gmail.com> wrote:
>>>>>>>
>>>>>>>> I checked the rc2, seems the TestHiveTableConcurrency is broken,
>>>>>>>> may need to fix it.
>>>>>>>>
>>>>>>>> 1. Download the tarball and check the signature & checksum: OK
>>>>>>>> 2. license checking: RAT checks passed.
>>>>>>>> 3. Build and test the project (java8):
>>>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency >
>>>>>>>> testConcurrentConnections FAILED
>>>>>>>> java.lang.AssertionError: Timeout
>>>>>>>> at org.junit.Assert.fail(Assert.java:88)
>>>>>>>> at org.junit.Assert.assertTrue(Assert.java:41)
>>>>>>>> at
>>>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
>>>>>>>>
>>>>>>>> On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <bl...@apache.org> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> I propose the following candidate to be released as the official
>>>>>>>>> Apache Iceberg 0.8.0-incubating release.
>>>>>>>>>
>>>>>>>>> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
>>>>>>>>> * This corresponds to the tag: apache-iceberg-0.8.0-incubating-rc2
>>>>>>>>> *
>>>>>>>>> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
>>>>>>>>> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5
>>>>>>>>>
>>>>>>>>> The release tarball, signature, and checksums are here:
>>>>>>>>> *
>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>>>>>>>
>>>>>>>>> You can find the KEYS file here:
>>>>>>>>> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
>>>>>>>>>
>>>>>>>>> Convenience binary artifacts are staged in Nexus. The Maven
>>>>>>>>> repository URL is:
>>>>>>>>> *
>>>>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
>>>>>>>>>
>>>>>>>>> This release contains many bug fixes and several new features:
>>>>>>>>> * Actions to remove orphaned files and to optimize metadata for
>>>>>>>>> query performance
>>>>>>>>> * Support for ORC data files
>>>>>>>>> * Snapshot cherry-picking
>>>>>>>>> * Incremental scan planning based on table history
>>>>>>>>> * In and notIn expressions
>>>>>>>>> * An InputFormat for writing MR jobs
>>>>>>>>>
>>>>>>>>> Please download, verify, and test.
>>>>>>>>>
>>>>>>>>> Please vote in the next 72 hours.
>>>>>>>>>
>>>>>>>>> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
>>>>>>>>> [ ] +0
>>>>>>>>> [ ] -1 Do not release this because...
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ryan Blue
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>
>> --
>> John Zhuge
>>
>
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by Samarth Jain <sa...@gmail.com>.
+1 (non-binding)
all checks passed
On Thu, Apr 30, 2020 at 4:06 PM John Zhuge <jz...@apache.org> wrote:
> +1 (non-binding)
>
> 1. Checked signature and checksum
> 2. Checked license
> 3. Built and ran unit tests.
>
>
> On Thu, Apr 30, 2020 at 2:24 PM Owen O'Malley <ow...@gmail.com>
> wrote:
>
>> +1
>>
>> 1. Checked signature and checksum
>> 2. Built and ran unit tests.
>> 3. Checked ORC version :)
>>
>> On Monday, ORC released 1.6.3, so we should grab those fixes soon.
>>
>> .. Owen
>>
>> On Thu, Apr 30, 2020 at 12:34 PM Dongjoon Hyun <do...@gmail.com>
>> wrote:
>>
>>> +1.
>>>
>>> 1. Verified checksum, sig, and license
>>> 3. Build from the source and run UTs.
>>> 4. Run some manual ORC write/read tests with Apache Spark 2.4.6-SNAPSHOT
>>> (as of today).
>>>
>>> Thank you, all!
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>> On Thu, Apr 30, 2020 at 10:28 AM parth brahmbhatt <
>>> brahmbhatt.parth@gmail.com> wrote:
>>>
>>>> +1. checks passed, did not observe the unit test failure.
>>>>
>>>> Thanks
>>>> Parth
>>>>
>>>> On Thu, Apr 30, 2020 at 9:13 AM Daniel Weeks <dw...@apache.org> wrote:
>>>>
>>>>> +1 all checks passed
>>>>>
>>>>> On Thu, Apr 30, 2020 at 8:53 AM Anton Okolnychyi
>>>>> <ao...@apple.com.invalid> wrote:
>>>>>
>>>>>> That test uses many concurrent writes and I’ve seen cases when it led
>>>>>> to deadlocks in our test HMS. I think HMS is capable of recovering on its
>>>>>> own but that process can be slow in highly concurrent environments. There
>>>>>> is a 2 min timeout in that test so it can potentially fail. I’ve seen a
>>>>>> deadlock but 2 min was always enough for that test in my local env and
>>>>>> internal/upstream build pipelines. If there is an environment that
>>>>>> constantly or frequently hits this problem, it would be great to check
>>>>>> debug logs.
>>>>>>
>>>>>> I am +1 on releasing RC2. I checked it locally.
>>>>>>
>>>>>> - Anton
>>>>>>
>>>>>> On 30 Apr 2020, at 02:52, Mass Dosage <ma...@gmail.com> wrote:
>>>>>>
>>>>>> The build for RC2 worked fine for me, I didn't get a failure on
>>>>>> "TestHiveTableConcurrency". Perhaps there is some kind of race condition in
>>>>>> the test? I have seen timeout errors like that when I ran tests on an
>>>>>> overloaded machine, could that have been the case?
>>>>>>
>>>>>> On Thu, 30 Apr 2020 at 08:32, OpenInx <op...@gmail.com> wrote:
>>>>>>
>>>>>>> I checked the rc2, seems the TestHiveTableConcurrency is broken, may
>>>>>>> need to fix it.
>>>>>>>
>>>>>>> 1. Download the tarball and check the signature & checksum: OK
>>>>>>> 2. license checking: RAT checks passed.
>>>>>>> 3. Build and test the project (java8):
>>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency >
>>>>>>> testConcurrentConnections FAILED
>>>>>>> java.lang.AssertionError: Timeout
>>>>>>> at org.junit.Assert.fail(Assert.java:88)
>>>>>>> at org.junit.Assert.assertTrue(Assert.java:41)
>>>>>>> at
>>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
>>>>>>>
>>>>>>> On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <bl...@apache.org> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> I propose the following candidate to be released as the official
>>>>>>>> Apache Iceberg 0.8.0-incubating release.
>>>>>>>>
>>>>>>>> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
>>>>>>>> * This corresponds to the tag: apache-iceberg-0.8.0-incubating-rc2
>>>>>>>> *
>>>>>>>> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
>>>>>>>> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5
>>>>>>>>
>>>>>>>> The release tarball, signature, and checksums are here:
>>>>>>>> *
>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>>>>>>
>>>>>>>> You can find the KEYS file here:
>>>>>>>> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
>>>>>>>>
>>>>>>>> Convenience binary artifacts are staged in Nexus. The Maven
>>>>>>>> repository URL is:
>>>>>>>> *
>>>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
>>>>>>>>
>>>>>>>> This release contains many bug fixes and several new features:
>>>>>>>> * Actions to remove orphaned files and to optimize metadata for
>>>>>>>> query performance
>>>>>>>> * Support for ORC data files
>>>>>>>> * Snapshot cherry-picking
>>>>>>>> * Incremental scan planning based on table history
>>>>>>>> * In and notIn expressions
>>>>>>>> * An InputFormat for writing MR jobs
>>>>>>>>
>>>>>>>> Please download, verify, and test.
>>>>>>>>
>>>>>>>> Please vote in the next 72 hours.
>>>>>>>>
>>>>>>>> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
>>>>>>>> [ ] +0
>>>>>>>> [ ] -1 Do not release this because...
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>>
>>>>>>>
>>>>>>
>
> --
> John Zhuge
>
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by John Zhuge <jz...@apache.org>.
+1 (non-binding)
1. Checked signature and checksum
2. Checked license
3. Built and ran unit tests.
On Thu, Apr 30, 2020 at 2:24 PM Owen O'Malley <ow...@gmail.com>
wrote:
> +1
>
> 1. Checked signature and checksum
> 2. Built and ran unit tests.
> 3. Checked ORC version :)
>
> On Monday, ORC released 1.6.3, so we should grab those fixes soon.
>
> .. Owen
>
> On Thu, Apr 30, 2020 at 12:34 PM Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> +1.
>>
>> 1. Verified checksum, sig, and license
>> 3. Build from the source and run UTs.
>> 4. Run some manual ORC write/read tests with Apache Spark 2.4.6-SNAPSHOT
>> (as of today).
>>
>> Thank you, all!
>>
>> Bests,
>> Dongjoon.
>>
>> On Thu, Apr 30, 2020 at 10:28 AM parth brahmbhatt <
>> brahmbhatt.parth@gmail.com> wrote:
>>
>>> +1. checks passed, did not observe the unit test failure.
>>>
>>> Thanks
>>> Parth
>>>
>>> On Thu, Apr 30, 2020 at 9:13 AM Daniel Weeks <dw...@apache.org> wrote:
>>>
>>>> +1 all checks passed
>>>>
>>>> On Thu, Apr 30, 2020 at 8:53 AM Anton Okolnychyi
>>>> <ao...@apple.com.invalid> wrote:
>>>>
>>>>> That test uses many concurrent writes and I’ve seen cases when it led
>>>>> to deadlocks in our test HMS. I think HMS is capable of recovering on its
>>>>> own but that process can be slow in highly concurrent environments. There
>>>>> is a 2 min timeout in that test so it can potentially fail. I’ve seen a
>>>>> deadlock but 2 min was always enough for that test in my local env and
>>>>> internal/upstream build pipelines. If there is an environment that
>>>>> constantly or frequently hits this problem, it would be great to check
>>>>> debug logs.
>>>>>
>>>>> I am +1 on releasing RC2. I checked it locally.
>>>>>
>>>>> - Anton
>>>>>
>>>>> On 30 Apr 2020, at 02:52, Mass Dosage <ma...@gmail.com> wrote:
>>>>>
>>>>> The build for RC2 worked fine for me, I didn't get a failure on
>>>>> "TestHiveTableConcurrency". Perhaps there is some kind of race condition in
>>>>> the test? I have seen timeout errors like that when I ran tests on an
>>>>> overloaded machine, could that have been the case?
>>>>>
>>>>> On Thu, 30 Apr 2020 at 08:32, OpenInx <op...@gmail.com> wrote:
>>>>>
>>>>>> I checked the rc2, seems the TestHiveTableConcurrency is broken, may
>>>>>> need to fix it.
>>>>>>
>>>>>> 1. Download the tarball and check the signature & checksum: OK
>>>>>> 2. license checking: RAT checks passed.
>>>>>> 3. Build and test the project (java8):
>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency >
>>>>>> testConcurrentConnections FAILED
>>>>>> java.lang.AssertionError: Timeout
>>>>>> at org.junit.Assert.fail(Assert.java:88)
>>>>>> at org.junit.Assert.assertTrue(Assert.java:41)
>>>>>> at
>>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
>>>>>>
>>>>>> On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <bl...@apache.org> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I propose the following candidate to be released as the official
>>>>>>> Apache Iceberg 0.8.0-incubating release.
>>>>>>>
>>>>>>> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
>>>>>>> * This corresponds to the tag: apache-iceberg-0.8.0-incubating-rc2
>>>>>>> *
>>>>>>> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
>>>>>>> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5
>>>>>>>
>>>>>>> The release tarball, signature, and checksums are here:
>>>>>>> *
>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>>>>>
>>>>>>> You can find the KEYS file here:
>>>>>>> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
>>>>>>>
>>>>>>> Convenience binary artifacts are staged in Nexus. The Maven
>>>>>>> repository URL is:
>>>>>>> *
>>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
>>>>>>>
>>>>>>> This release contains many bug fixes and several new features:
>>>>>>> * Actions to remove orphaned files and to optimize metadata for
>>>>>>> query performance
>>>>>>> * Support for ORC data files
>>>>>>> * Snapshot cherry-picking
>>>>>>> * Incremental scan planning based on table history
>>>>>>> * In and notIn expressions
>>>>>>> * An InputFormat for writing MR jobs
>>>>>>>
>>>>>>> Please download, verify, and test.
>>>>>>>
>>>>>>> Please vote in the next 72 hours.
>>>>>>>
>>>>>>> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
>>>>>>> [ ] +0
>>>>>>> [ ] -1 Do not release this because...
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>>
>>>>>>
>>>>>
--
John Zhuge
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by Owen O'Malley <ow...@gmail.com>.
+1
1. Checked signature and checksum
2. Built and ran unit tests.
3. Checked ORC version :)
On Monday, ORC released 1.6.3, so we should grab those fixes soon.
.. Owen
On Thu, Apr 30, 2020 at 12:34 PM Dongjoon Hyun <do...@gmail.com>
wrote:
> +1.
>
> 1. Verified checksum, sig, and license
> 3. Build from the source and run UTs.
> 4. Run some manual ORC write/read tests with Apache Spark 2.4.6-SNAPSHOT
> (as of today).
>
> Thank you, all!
>
> Bests,
> Dongjoon.
>
> On Thu, Apr 30, 2020 at 10:28 AM parth brahmbhatt <
> brahmbhatt.parth@gmail.com> wrote:
>
>> +1. checks passed, did not observe the unit test failure.
>>
>> Thanks
>> Parth
>>
>> On Thu, Apr 30, 2020 at 9:13 AM Daniel Weeks <dw...@apache.org> wrote:
>>
>>> +1 all checks passed
>>>
>>> On Thu, Apr 30, 2020 at 8:53 AM Anton Okolnychyi
>>> <ao...@apple.com.invalid> wrote:
>>>
>>>> That test uses many concurrent writes and I’ve seen cases when it led
>>>> to deadlocks in our test HMS. I think HMS is capable of recovering on its
>>>> own but that process can be slow in highly concurrent environments. There
>>>> is a 2 min timeout in that test so it can potentially fail. I’ve seen a
>>>> deadlock but 2 min was always enough for that test in my local env and
>>>> internal/upstream build pipelines. If there is an environment that
>>>> constantly or frequently hits this problem, it would be great to check
>>>> debug logs.
>>>>
>>>> I am +1 on releasing RC2. I checked it locally.
>>>>
>>>> - Anton
>>>>
>>>> On 30 Apr 2020, at 02:52, Mass Dosage <ma...@gmail.com> wrote:
>>>>
>>>> The build for RC2 worked fine for me, I didn't get a failure on
>>>> "TestHiveTableConcurrency". Perhaps there is some kind of race condition in
>>>> the test? I have seen timeout errors like that when I ran tests on an
>>>> overloaded machine, could that have been the case?
>>>>
>>>> On Thu, 30 Apr 2020 at 08:32, OpenInx <op...@gmail.com> wrote:
>>>>
>>>>> I checked the rc2, seems the TestHiveTableConcurrency is broken, may
>>>>> need to fix it.
>>>>>
>>>>> 1. Download the tarball and check the signature & checksum: OK
>>>>> 2. license checking: RAT checks passed.
>>>>> 3. Build and test the project (java8):
>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency >
>>>>> testConcurrentConnections FAILED
>>>>> java.lang.AssertionError: Timeout
>>>>> at org.junit.Assert.fail(Assert.java:88)
>>>>> at org.junit.Assert.assertTrue(Assert.java:41)
>>>>> at
>>>>> org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
>>>>>
>>>>> On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <bl...@apache.org> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> I propose the following candidate to be released as the official
>>>>>> Apache Iceberg 0.8.0-incubating release.
>>>>>>
>>>>>> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
>>>>>> * This corresponds to the tag: apache-iceberg-0.8.0-incubating-rc2
>>>>>> *
>>>>>> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
>>>>>> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5
>>>>>>
>>>>>> The release tarball, signature, and checksums are here:
>>>>>> *
>>>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>>>>
>>>>>> You can find the KEYS file here:
>>>>>> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
>>>>>>
>>>>>> Convenience binary artifacts are staged in Nexus. The Maven
>>>>>> repository URL is:
>>>>>> *
>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
>>>>>>
>>>>>> This release contains many bug fixes and several new features:
>>>>>> * Actions to remove orphaned files and to optimize metadata for query
>>>>>> performance
>>>>>> * Support for ORC data files
>>>>>> * Snapshot cherry-picking
>>>>>> * Incremental scan planning based on table history
>>>>>> * In and notIn expressions
>>>>>> * An InputFormat for writing MR jobs
>>>>>>
>>>>>> Please download, verify, and test.
>>>>>>
>>>>>> Please vote in the next 72 hours.
>>>>>>
>>>>>> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
>>>>>> [ ] +0
>>>>>> [ ] -1 Do not release this because...
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>>
>>>>>
>>>>
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by Dongjoon Hyun <do...@gmail.com>.
+1.
1. Verified checksum, sig, and license
3. Build from the source and run UTs.
4. Run some manual ORC write/read tests with Apache Spark 2.4.6-SNAPSHOT
(as of today).
Thank you, all!
Bests,
Dongjoon.
On Thu, Apr 30, 2020 at 10:28 AM parth brahmbhatt <
brahmbhatt.parth@gmail.com> wrote:
> +1. checks passed, did not observe the unit test failure.
>
> Thanks
> Parth
>
> On Thu, Apr 30, 2020 at 9:13 AM Daniel Weeks <dw...@apache.org> wrote:
>
>> +1 all checks passed
>>
>> On Thu, Apr 30, 2020 at 8:53 AM Anton Okolnychyi
>> <ao...@apple.com.invalid> wrote:
>>
>>> That test uses many concurrent writes and I’ve seen cases when it led to
>>> deadlocks in our test HMS. I think HMS is capable of recovering on its own
>>> but that process can be slow in highly concurrent environments. There is a
>>> 2 min timeout in that test so it can potentially fail. I’ve seen a deadlock
>>> but 2 min was always enough for that test in my local env and
>>> internal/upstream build pipelines. If there is an environment that
>>> constantly or frequently hits this problem, it would be great to check
>>> debug logs.
>>>
>>> I am +1 on releasing RC2. I checked it locally.
>>>
>>> - Anton
>>>
>>> On 30 Apr 2020, at 02:52, Mass Dosage <ma...@gmail.com> wrote:
>>>
>>> The build for RC2 worked fine for me, I didn't get a failure on
>>> "TestHiveTableConcurrency". Perhaps there is some kind of race condition in
>>> the test? I have seen timeout errors like that when I ran tests on an
>>> overloaded machine, could that have been the case?
>>>
>>> On Thu, 30 Apr 2020 at 08:32, OpenInx <op...@gmail.com> wrote:
>>>
>>>> I checked the rc2, seems the TestHiveTableConcurrency is broken, may
>>>> need to fix it.
>>>>
>>>> 1. Download the tarball and check the signature & checksum: OK
>>>> 2. license checking: RAT checks passed.
>>>> 3. Build and test the project (java8):
>>>> org.apache.iceberg.hive.TestHiveTableConcurrency >
>>>> testConcurrentConnections FAILED
>>>> java.lang.AssertionError: Timeout
>>>> at org.junit.Assert.fail(Assert.java:88)
>>>> at org.junit.Assert.assertTrue(Assert.java:41)
>>>> at
>>>> org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
>>>>
>>>> On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <bl...@apache.org> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I propose the following candidate to be released as the official
>>>>> Apache Iceberg 0.8.0-incubating release.
>>>>>
>>>>> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
>>>>> * This corresponds to the tag: apache-iceberg-0.8.0-incubating-rc2
>>>>> *
>>>>> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
>>>>> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5
>>>>>
>>>>> The release tarball, signature, and checksums are here:
>>>>> *
>>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>>>
>>>>> You can find the KEYS file here:
>>>>> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
>>>>>
>>>>> Convenience binary artifacts are staged in Nexus. The Maven repository
>>>>> URL is:
>>>>> *
>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
>>>>>
>>>>> This release contains many bug fixes and several new features:
>>>>> * Actions to remove orphaned files and to optimize metadata for query
>>>>> performance
>>>>> * Support for ORC data files
>>>>> * Snapshot cherry-picking
>>>>> * Incremental scan planning based on table history
>>>>> * In and notIn expressions
>>>>> * An InputFormat for writing MR jobs
>>>>>
>>>>> Please download, verify, and test.
>>>>>
>>>>> Please vote in the next 72 hours.
>>>>>
>>>>> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
>>>>> [ ] +0
>>>>> [ ] -1 Do not release this because...
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>>
>>>>
>>>
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by parth brahmbhatt <br...@gmail.com>.
+1. checks passed, did not observe the unit test failure.
Thanks
Parth
On Thu, Apr 30, 2020 at 9:13 AM Daniel Weeks <dw...@apache.org> wrote:
> +1 all checks passed
>
> On Thu, Apr 30, 2020 at 8:53 AM Anton Okolnychyi
> <ao...@apple.com.invalid> wrote:
>
>> That test uses many concurrent writes and I’ve seen cases when it led to
>> deadlocks in our test HMS. I think HMS is capable of recovering on its own
>> but that process can be slow in highly concurrent environments. There is a
>> 2 min timeout in that test so it can potentially fail. I’ve seen a deadlock
>> but 2 min was always enough for that test in my local env and
>> internal/upstream build pipelines. If there is an environment that
>> constantly or frequently hits this problem, it would be great to check
>> debug logs.
>>
>> I am +1 on releasing RC2. I checked it locally.
>>
>> - Anton
>>
>> On 30 Apr 2020, at 02:52, Mass Dosage <ma...@gmail.com> wrote:
>>
>> The build for RC2 worked fine for me, I didn't get a failure on
>> "TestHiveTableConcurrency". Perhaps there is some kind of race condition in
>> the test? I have seen timeout errors like that when I ran tests on an
>> overloaded machine, could that have been the case?
>>
>> On Thu, 30 Apr 2020 at 08:32, OpenInx <op...@gmail.com> wrote:
>>
>>> I checked the rc2, seems the TestHiveTableConcurrency is broken, may
>>> need to fix it.
>>>
>>> 1. Download the tarball and check the signature & checksum: OK
>>> 2. license checking: RAT checks passed.
>>> 3. Build and test the project (java8):
>>> org.apache.iceberg.hive.TestHiveTableConcurrency >
>>> testConcurrentConnections FAILED
>>> java.lang.AssertionError: Timeout
>>> at org.junit.Assert.fail(Assert.java:88)
>>> at org.junit.Assert.assertTrue(Assert.java:41)
>>> at
>>> org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
>>>
>>> On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <bl...@apache.org> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I propose the following candidate to be released as the official Apache
>>>> Iceberg 0.8.0-incubating release.
>>>>
>>>> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
>>>> * This corresponds to the tag: apache-iceberg-0.8.0-incubating-rc2
>>>> *
>>>> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
>>>> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5
>>>>
>>>> The release tarball, signature, and checksums are here:
>>>> *
>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>>
>>>> You can find the KEYS file here:
>>>> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
>>>>
>>>> Convenience binary artifacts are staged in Nexus. The Maven repository
>>>> URL is:
>>>> *
>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
>>>>
>>>> This release contains many bug fixes and several new features:
>>>> * Actions to remove orphaned files and to optimize metadata for query
>>>> performance
>>>> * Support for ORC data files
>>>> * Snapshot cherry-picking
>>>> * Incremental scan planning based on table history
>>>> * In and notIn expressions
>>>> * An InputFormat for writing MR jobs
>>>>
>>>> Please download, verify, and test.
>>>>
>>>> Please vote in the next 72 hours.
>>>>
>>>> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
>>>> [ ] +0
>>>> [ ] -1 Do not release this because...
>>>>
>>>> --
>>>> Ryan Blue
>>>>
>>>
>>
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by Daniel Weeks <dw...@apache.org>.
+1 all checks passed
On Thu, Apr 30, 2020 at 8:53 AM Anton Okolnychyi
<ao...@apple.com.invalid> wrote:
> That test uses many concurrent writes and I’ve seen cases when it led to
> deadlocks in our test HMS. I think HMS is capable of recovering on its own
> but that process can be slow in highly concurrent environments. There is a
> 2 min timeout in that test so it can potentially fail. I’ve seen a deadlock
> but 2 min was always enough for that test in my local env and
> internal/upstream build pipelines. If there is an environment that
> constantly or frequently hits this problem, it would be great to check
> debug logs.
>
> I am +1 on releasing RC2. I checked it locally.
>
> - Anton
>
> On 30 Apr 2020, at 02:52, Mass Dosage <ma...@gmail.com> wrote:
>
> The build for RC2 worked fine for me, I didn't get a failure on
> "TestHiveTableConcurrency". Perhaps there is some kind of race condition in
> the test? I have seen timeout errors like that when I ran tests on an
> overloaded machine, could that have been the case?
>
> On Thu, 30 Apr 2020 at 08:32, OpenInx <op...@gmail.com> wrote:
>
>> I checked the rc2, seems the TestHiveTableConcurrency is broken, may need
>> to fix it.
>>
>> 1. Download the tarball and check the signature & checksum: OK
>> 2. license checking: RAT checks passed.
>> 3. Build and test the project (java8):
>> org.apache.iceberg.hive.TestHiveTableConcurrency >
>> testConcurrentConnections FAILED
>> java.lang.AssertionError: Timeout
>> at org.junit.Assert.fail(Assert.java:88)
>> at org.junit.Assert.assertTrue(Assert.java:41)
>> at
>> org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
>>
>> On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <bl...@apache.org> wrote:
>>
>>> Hi everyone,
>>>
>>> I propose the following candidate to be released as the official Apache
>>> Iceberg 0.8.0-incubating release.
>>>
>>> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
>>> * This corresponds to the tag: apache-iceberg-0.8.0-incubating-rc2
>>> *
>>> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
>>> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5
>>>
>>> The release tarball, signature, and checksums are here:
>>> *
>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>>
>>> You can find the KEYS file here:
>>> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
>>>
>>> Convenience binary artifacts are staged in Nexus. The Maven repository
>>> URL is:
>>> *
>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
>>>
>>> This release contains many bug fixes and several new features:
>>> * Actions to remove orphaned files and to optimize metadata for query
>>> performance
>>> * Support for ORC data files
>>> * Snapshot cherry-picking
>>> * Incremental scan planning based on table history
>>> * In and notIn expressions
>>> * An InputFormat for writing MR jobs
>>>
>>> Please download, verify, and test.
>>>
>>> Please vote in the next 72 hours.
>>>
>>> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
>>> [ ] +0
>>> [ ] -1 Do not release this because...
>>>
>>> --
>>> Ryan Blue
>>>
>>
>
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by Anton Okolnychyi <ao...@apple.com.INVALID>.
That test uses many concurrent writes and I’ve seen cases when it led to deadlocks in our test HMS. I think HMS is capable of recovering on its own but that process can be slow in highly concurrent environments. There is a 2 min timeout in that test so it can potentially fail. I’ve seen a deadlock but 2 min was always enough for that test in my local env and internal/upstream build pipelines. If there is an environment that constantly or frequently hits this problem, it would be great to check debug logs.
I am +1 on releasing RC2. I checked it locally.
- Anton
> On 30 Apr 2020, at 02:52, Mass Dosage <ma...@gmail.com> wrote:
>
> The build for RC2 worked fine for me, I didn't get a failure on "TestHiveTableConcurrency". Perhaps there is some kind of race condition in the test? I have seen timeout errors like that when I ran tests on an overloaded machine, could that have been the case?
>
> On Thu, 30 Apr 2020 at 08:32, OpenInx <openinx@gmail.com <ma...@gmail.com>> wrote:
> I checked the rc2, seems the TestHiveTableConcurrency is broken, may need to fix it.
>
> 1. Download the tarball and check the signature & checksum: OK
> 2. license checking: RAT checks passed.
> 3. Build and test the project (java8):
> org.apache.iceberg.hive.TestHiveTableConcurrency > testConcurrentConnections FAILED
> java.lang.AssertionError: Timeout
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
>
> On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <blue@apache.org <ma...@apache.org>> wrote:
> Hi everyone,
>
> I propose the following candidate to be released as the official Apache Iceberg 0.8.0-incubating release.
>
> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
> * This corresponds to the tag: apache-iceberg-0.8.0-incubating-rc2
> * https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2 <https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2>
> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5 <https://github.com/apache/incubator-iceberg/tree/8c05a2f5>
>
> The release tarball, signature, and checksums are here:
> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/ <https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/>
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS <https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS>
>
> Convenience binary artifacts are staged in Nexus. The Maven repository URL is:
> * https://repository.apache.org/content/repositories/orgapacheiceberg-1006/ <https://repository.apache.org/content/repositories/orgapacheiceberg-1006/>
>
> This release contains many bug fixes and several new features:
> * Actions to remove orphaned files and to optimize metadata for query performance
> * Support for ORC data files
> * Snapshot cherry-picking
> * Incremental scan planning based on table history
> * In and notIn expressions
> * An InputFormat for writing MR jobs
>
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
> [ ] +0
> [ ] -1 Do not release this because...
>
> --
> Ryan Blue
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by Mass Dosage <ma...@gmail.com>.
The build for RC2 worked fine for me, I didn't get a failure on
"TestHiveTableConcurrency". Perhaps there is some kind of race condition in
the test? I have seen timeout errors like that when I ran tests on an
overloaded machine, could that have been the case?
On Thu, 30 Apr 2020 at 08:32, OpenInx <op...@gmail.com> wrote:
> I checked the rc2, seems the TestHiveTableConcurrency is broken, may need
> to fix it.
>
> 1. Download the tarball and check the signature & checksum: OK
> 2. license checking: RAT checks passed.
> 3. Build and test the project (java8):
> org.apache.iceberg.hive.TestHiveTableConcurrency >
> testConcurrentConnections FAILED
> java.lang.AssertionError: Timeout
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
>
> On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <bl...@apache.org> wrote:
>
>> Hi everyone,
>>
>> I propose the following candidate to be released as the official Apache
>> Iceberg 0.8.0-incubating release.
>>
>> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
>> * This corresponds to the tag: apache-iceberg-0.8.0-incubating-rc2
>> *
>> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
>> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5
>>
>> The release tarball, signature, and checksums are here:
>> *
>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>>
>> You can find the KEYS file here:
>> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
>>
>> Convenience binary artifacts are staged in Nexus. The Maven repository
>> URL is:
>> *
>> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
>>
>> This release contains many bug fixes and several new features:
>> * Actions to remove orphaned files and to optimize metadata for query
>> performance
>> * Support for ORC data files
>> * Snapshot cherry-picking
>> * Incremental scan planning based on table history
>> * In and notIn expressions
>> * An InputFormat for writing MR jobs
>>
>> Please download, verify, and test.
>>
>> Please vote in the next 72 hours.
>>
>> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
>> [ ] +0
>> [ ] -1 Do not release this because...
>>
>> --
>> Ryan Blue
>>
>
Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Posted by OpenInx <op...@gmail.com>.
I checked the rc2, seems the TestHiveTableConcurrency is broken, may need
to fix it.
1. Download the tarball and check the signature & checksum: OK
2. license checking: RAT checks passed.
3. Build and test the project (java8):
org.apache.iceberg.hive.TestHiveTableConcurrency >
testConcurrentConnections FAILED
java.lang.AssertionError: Timeout
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.iceberg.hive.TestHiveTableConcurrency.testConcurrentConnections(TestHiveTableConcurrency.java:106)
On Thu, Apr 30, 2020 at 9:29 AM Ryan Blue <bl...@apache.org> wrote:
> Hi everyone,
>
> I propose the following candidate to be released as the official Apache
> Iceberg 0.8.0-incubating release.
>
> The commit id is 8c05a2f5f1c8b111c049d43cf15cd8a51920dda1
> * This corresponds to the tag: apache-iceberg-0.8.0-incubating-rc2
> *
> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2
> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5
>
> The release tarball, signature, and checksums are here:
> *
> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS
>
> Convenience binary artifacts are staged in Nexus. The Maven repository URL
> is:
> *
> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/
>
> This release contains many bug fixes and several new features:
> * Actions to remove orphaned files and to optimize metadata for query
> performance
> * Support for ORC data files
> * Snapshot cherry-picking
> * Incremental scan planning based on table history
> * In and notIn expressions
> * An InputFormat for writing MR jobs
>
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating
> [ ] +0
> [ ] -1 Do not release this because...
>
> --
> Ryan Blue
>