You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Gabor Kaszab <ga...@apache.org> on 2022/11/17 10:30:19 UTC
[VOTE] Release Apache Iceberg 1.1.0 RC2
Hi Everyone,
I propose that we release the following RC as the official Apache
Iceberg 1.1.0 release.
The commit ID is b3eaf0c6cb9cf6357a925c7443baadd54515a971
* This corresponds to the tag: apache-iceberg-1.1.0-rc2
* https://github.com/apache/iceberg/commits/apache-iceberg-1.1.0-rc2
* https://github.com/apache/iceberg/tree/b3eaf0c6cb9cf6357a925c7443baadd54515a971
The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-1.1.0-rc2
You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/iceberg/KEYS
Convenience binary artifacts are staged on Nexus. The Maven repository URL is:
* https://repository.apache.org/content/repositories/orgapacheiceberg-1111/
Please download, verify, and test.
Please vote in the next 72 hours.
[ ] +1 Release this as Apache Iceberg 1.1.0
[ ] +0
[ ] -1 Do not release this because...
Again, thanks to *Fokko* for running the RC creation steps for me!
Re: [VOTE] Release Apache Iceberg 1.1.0 RC2
Posted by Gabor Kaszab <ga...@apache.org>.
Thanks for spotting this break and giving a fix, Fokko!
Let me start another thread for the new RC.
Gabor
On Sun, Nov 20, 2022 at 9:27 PM Driesprong, Fokko <fo...@apache.org> wrote:
> Thanks! Would be great to get this one in before RC3 as well:
> https://github.com/apache/iceberg/pull/6195
>
> Op zo 20 nov. 2022 om 21:03 schreef Ryan Blue <bl...@tabular.io>:
>
>> I just merged Fokko's fix for the breaking change, so we should be
>> unblocked now. Thanks, Fokko!
>>
>> On Sat, Nov 19, 2022 at 3:30 PM Ryan Blue <bl...@tabular.io> wrote:
>>
>>> Thanks, Fokko! I just reviewed the PR and it's almost ready to go.
>>>
>>> On Sat, Nov 19, 2022 at 3:01 PM Driesprong, Fokko <fo...@apache.org>
>>> wrote:
>>>
>>>> Hey everyone,
>>>>
>>>> Wanted to let you know that I got a working fix for the breaking API:
>>>> https://github.com/apache/iceberg/pull/6220 Let me know what you think.
>>>>
>>>> Kind regards,
>>>> Fokko Driesprong
>>>>
>>>> Op za 19 nov. 2022 om 14:20 schreef leilei hu <hu...@gmail.com>:
>>>>
>>>>> +1(non-binding)
>>>>> verified(java 8):
>>>>>
>>>>> - Create table using HiveCatalog and HadoopCatalog
>>>>> - Spark Structured Streaming with Spark 3.2.1
>>>>> - Spark query with Spark’s DataSourceV2 API
>>>>> - Ran build with JDK8
>>>>>
>>>>> 2022年11月18日 上午12:39，Szehon Ho <sz...@gmail.com> 写道：
>>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> Ryan Blue
>>> Tabular
>>>
>>
>>
>> --
>> Ryan Blue
>> Tabular
>>
>
Re: [VOTE] Release Apache Iceberg 1.1.0 RC2
Posted by "Driesprong, Fokko" <fo...@apache.org>.
Thanks! Would be great to get this one in before RC3 as well:
https://github.com/apache/iceberg/pull/6195
Op zo 20 nov. 2022 om 21:03 schreef Ryan Blue <bl...@tabular.io>:
> I just merged Fokko's fix for the breaking change, so we should be
> unblocked now. Thanks, Fokko!
>
> On Sat, Nov 19, 2022 at 3:30 PM Ryan Blue <bl...@tabular.io> wrote:
>
>> Thanks, Fokko! I just reviewed the PR and it's almost ready to go.
>>
>> On Sat, Nov 19, 2022 at 3:01 PM Driesprong, Fokko <fo...@apache.org>
>> wrote:
>>
>>> Hey everyone,
>>>
>>> Wanted to let you know that I got a working fix for the breaking API:
>>> https://github.com/apache/iceberg/pull/6220 Let me know what you think.
>>>
>>> Kind regards,
>>> Fokko Driesprong
>>>
>>> Op za 19 nov. 2022 om 14:20 schreef leilei hu <hu...@gmail.com>:
>>>
>>>> +1(non-binding)
>>>> verified(java 8):
>>>>
>>>> - Create table using HiveCatalog and HadoopCatalog
>>>> - Spark Structured Streaming with Spark 3.2.1
>>>> - Spark query with Spark’s DataSourceV2 API
>>>> - Ran build with JDK8
>>>>
>>>> 2022年11月18日 上午12:39，Szehon Ho <sz...@gmail.com> 写道：
>>>>
>>>> +1 (non-binding)
>>>>
>>>>
>>>>
>>
>> --
>> Ryan Blue
>> Tabular
>>
>
>
> --
> Ryan Blue
> Tabular
>
Re: [VOTE] Release Apache Iceberg 1.1.0 RC2
Posted by Ryan Blue <bl...@tabular.io>.
I just merged Fokko's fix for the breaking change, so we should be
unblocked now. Thanks, Fokko!
On Sat, Nov 19, 2022 at 3:30 PM Ryan Blue <bl...@tabular.io> wrote:
> Thanks, Fokko! I just reviewed the PR and it's almost ready to go.
>
> On Sat, Nov 19, 2022 at 3:01 PM Driesprong, Fokko <fo...@apache.org>
> wrote:
>
>> Hey everyone,
>>
>> Wanted to let you know that I got a working fix for the breaking API:
>> https://github.com/apache/iceberg/pull/6220 Let me know what you think.
>>
>> Kind regards,
>> Fokko Driesprong
>>
>> Op za 19 nov. 2022 om 14:20 schreef leilei hu <hu...@gmail.com>:
>>
>>> +1(non-binding)
>>> verified(java 8):
>>>
>>> - Create table using HiveCatalog and HadoopCatalog
>>> - Spark Structured Streaming with Spark 3.2.1
>>> - Spark query with Spark’s DataSourceV2 API
>>> - Ran build with JDK8
>>>
>>> 2022年11月18日 上午12:39，Szehon Ho <sz...@gmail.com> 写道：
>>>
>>> +1 (non-binding)
>>>
>>>
>>>
>
> --
> Ryan Blue
> Tabular
>
--
Ryan Blue
Tabular
Re: [VOTE] Release Apache Iceberg 1.1.0 RC2
Posted by Ryan Blue <bl...@tabular.io>.
Thanks, Fokko! I just reviewed the PR and it's almost ready to go.
On Sat, Nov 19, 2022 at 3:01 PM Driesprong, Fokko <fo...@apache.org> wrote:
> Hey everyone,
>
> Wanted to let you know that I got a working fix for the breaking API:
> https://github.com/apache/iceberg/pull/6220 Let me know what you think.
>
> Kind regards,
> Fokko Driesprong
>
> Op za 19 nov. 2022 om 14:20 schreef leilei hu <hu...@gmail.com>:
>
>> +1(non-binding)
>> verified(java 8):
>>
>> - Create table using HiveCatalog and HadoopCatalog
>> - Spark Structured Streaming with Spark 3.2.1
>> - Spark query with Spark’s DataSourceV2 API
>> - Ran build with JDK8
>>
>> 2022年11月18日 上午12:39，Szehon Ho <sz...@gmail.com> 写道：
>>
>> +1 (non-binding)
>>
>>
>>
--
Ryan Blue
Tabular
Re: [VOTE] Release Apache Iceberg 1.1.0 RC2
Posted by "Driesprong, Fokko" <fo...@apache.org>.
Hey everyone,
Wanted to let you know that I got a working fix for the breaking API:
https://github.com/apache/iceberg/pull/6220 Let me know what you think.
Kind regards,
Fokko Driesprong
Op za 19 nov. 2022 om 14:20 schreef leilei hu <hu...@gmail.com>:
> +1(non-binding)
> verified(java 8):
>
> - Create table using HiveCatalog and HadoopCatalog
> - Spark Structured Streaming with Spark 3.2.1
> - Spark query with Spark’s DataSourceV2 API
> - Ran build with JDK8
>
> 2022年11月18日 上午12:39，Szehon Ho <sz...@gmail.com> 写道：
>
> +1 (non-binding)
>
>
>
Re: [VOTE] Release Apache Iceberg 1.1.0 RC2
Posted by leilei hu <hu...@gmail.com>.
+1(non-binding)
verified(java 8):
- Create table using HiveCatalog and HadoopCatalog
- Spark Structured Streaming with Spark 3.2.1
- Spark query with Spark’s DataSourceV2 API
- Ran build with JDK8
> 2022年11月18日 上午12:39，Szehon Ho <sz...@gmail.com> 写道：
>
> +1 (non-binding)
Re: [VOTE] Release Apache Iceberg 1.1.0 RC2
Posted by Ajantha Bhat <aj...@gmail.com>.
@Szehon:
This failure looks to be the same as #6060.
I did report two flaky tests 3 weeks ago. But we didn't investigate it I
guess.
https://github.com/apache/iceberg/issues/6060
https://github.com/apache/iceberg/issues/5986
Thanks,
Ajantha
On Thu, Nov 17, 2022 at 10:09 PM Szehon Ho <sz...@gmail.com> wrote:
> +1 (non-binding)
> 1. Verify signature
> 2. Verify checksum
> 3. License RAT check
>
>
> 4. Run unit test, Actually got a failure:
>
> org.apache.iceberg.spark.extensions.TestCopyOnWriteDelete >
> testDeleteWithSnapshotIsolation[catalogName = spark_catalog, implementation
> = org.apache.iceberg.spark.SparkSessionCatalog, config = {type=hive,
> default-namespace=default, clients=1, parquet-enabled=false,
> cache-enabled=false}, format = avro, vectorized = false, distributionMode =
> range] FAILED
> java.util.concurrent.ExecutionException:
> java.lang.IllegalStateException: Runtime file filtering is not possible:
> the table has been concurrently modified. Row-level operation scan snapshot
> ID: 3904024863143561157, current table snapshot ID: 6400925136398165989. If
> multiple threads modify the table, use independent Spark sessions in each
> thread.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.iceberg.spark.extensions.TestDelete.testDeleteWithSnapshotIsolation(TestDelete.java:873)
>
> But I think its a concurrency issue and not worth stopping RC for (will
> file a issue)
>
> 5. Check with spark 3.3, create table and query data, metadata tables.
>
> The convenience binaries
> https://repository.apache.org/content/repositories/orgapacheiceberg-1111/org/apache/iceberg/
> have spark 3.2 jar but not spark 3.3, is that intentional? Anyway
> built spark 3.3 runtime to verify, and works.
>
> Thanks
> Szehon
>
> On Thu, Nov 17, 2022 at 11:30 AM Gabor Kaszab <ga...@apache.org>
> wrote:
>
>> Hi Everyone,
>>
>> I propose that we release the following RC as the official Apache Iceberg 1.1.0 release.
>>
>> The commit ID is b3eaf0c6cb9cf6357a925c7443baadd54515a971
>> * This corresponds to the tag: apache-iceberg-1.1.0-rc2
>> * https://github.com/apache/iceberg/commits/apache-iceberg-1.1.0-rc2
>> * https://github.com/apache/iceberg/tree/b3eaf0c6cb9cf6357a925c7443baadd54515a971
>>
>> The release tarball, signature, and checksums are here:
>> * https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-1.1.0-rc2
>>
>> You can find the KEYS file here:
>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>
>> Convenience binary artifacts are staged on Nexus. The Maven repository URL is:
>> * https://repository.apache.org/content/repositories/orgapacheiceberg-1111/
>>
>> Please download, verify, and test.
>>
>> Please vote in the next 72 hours.
>>
>> [ ] +1 Release this as Apache Iceberg 1.1.0
>> [ ] +0
>> [ ] -1 Do not release this because...
>>
>> Again, thanks to *Fokko* for running the RC creation steps for me!
>>
>>
Re: [VOTE] Release Apache Iceberg 1.1.0 RC2
Posted by Szehon Ho <sz...@gmail.com>.
+1 (non-binding)
1. Verify signature
2. Verify checksum
3. License RAT check
4. Run unit test, Actually got a failure:
org.apache.iceberg.spark.extensions.TestCopyOnWriteDelete >
testDeleteWithSnapshotIsolation[catalogName = spark_catalog, implementation
= org.apache.iceberg.spark.SparkSessionCatalog, config = {type=hive,
default-namespace=default, clients=1, parquet-enabled=false,
cache-enabled=false}, format = avro, vectorized = false, distributionMode =
range] FAILED
java.util.concurrent.ExecutionException:
java.lang.IllegalStateException: Runtime file filtering is not possible:
the table has been concurrently modified. Row-level operation scan snapshot
ID: 3904024863143561157, current table snapshot ID: 6400925136398165989. If
multiple threads modify the table, use independent Spark sessions in each
thread.
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.iceberg.spark.extensions.TestDelete.testDeleteWithSnapshotIsolation(TestDelete.java:873)
But I think its a concurrency issue and not worth stopping RC for (will
file a issue)
5. Check with spark 3.3, create table and query data, metadata tables.
The convenience binaries
https://repository.apache.org/content/repositories/orgapacheiceberg-1111/org/apache/iceberg/
have spark 3.2 jar but not spark 3.3, is that intentional? Anyway
built spark 3.3 runtime to verify, and works.
Thanks
Szehon
On Thu, Nov 17, 2022 at 11:30 AM Gabor Kaszab <ga...@apache.org>
wrote:
> Hi Everyone,
>
> I propose that we release the following RC as the official Apache Iceberg 1.1.0 release.
>
> The commit ID is b3eaf0c6cb9cf6357a925c7443baadd54515a971
> * This corresponds to the tag: apache-iceberg-1.1.0-rc2
> * https://github.com/apache/iceberg/commits/apache-iceberg-1.1.0-rc2
> * https://github.com/apache/iceberg/tree/b3eaf0c6cb9cf6357a925c7443baadd54515a971
>
> The release tarball, signature, and checksums are here:
> * https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-1.1.0-rc2
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>
> Convenience binary artifacts are staged on Nexus. The Maven repository URL is:
> * https://repository.apache.org/content/repositories/orgapacheiceberg-1111/
>
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as Apache Iceberg 1.1.0
> [ ] +0
> [ ] -1 Do not release this because...
>
> Again, thanks to *Fokko* for running the RC creation steps for me!
>
>
Re: [VOTE] Release Apache Iceberg 1.1.0 RC2
Posted by "Driesprong, Fokko" <fo...@apache.org>.
-1 (non-binding)
For testing the release I plugged in the latest RC2 in Trino and found out
that we have regression: https://github.com/trinodb/trino/pull/15079/files
It throws this exception:
java.lang.UnsupportedOperationException: hash(value) is not supported on
the base Bucket class
at org.apache.iceberg.transforms.Bucket.hash(Bucket.java:90)
at org.apache.iceberg.transforms.Bucket.apply(Bucket.java:99)
at org.apache.iceberg.transforms.Bucket.apply(Bucket.java:38)
at
io.trino.plugin.iceberg.IcebergPageSink.applyTransform(IcebergPageSink.java:372)
at
io.trino.plugin.iceberg.IcebergPageSink.getPartitionData(IcebergPageSink.java:364)
at
io.trino.plugin.iceberg.IcebergPageSink.getWriterIndexes(IcebergPageSink.java:288)
at
io.trino.plugin.iceberg.IcebergPageSink.writePage(IcebergPageSink.java:215)
at
io.trino.plugin.iceberg.IcebergPageSink.doAppend(IcebergPageSink.java:210)
at
io.trino.plugin.iceberg.IcebergPageSink.appendPage(IcebergPageSink.java:161)
We've removed the source type from the transform
<https://github.com/apache/iceberg/pull/5601>, so we can do lazy binding
which is very nice, but this wasn't deprecated in 1.0.0
<https://github.com/apache/iceberg/blob/1.0.x/api/src/main/java/org/apache/iceberg/transforms/Transform.java#L42>.
I'm not sure if we can make this backward compatible without diving into
the details, mostly because we've also built on top of it
<https://github.com/apache/iceberg/pull/5707>.
Best, Fokko
Op do 17 nov. 2022 om 18:16 schreef Eduard Tudenhoefner <ed...@tabular.io>:
> +1 (non-binding)
>
> - validated checksum and signature
> - checked license docs & ran RAT checks
> - ran build and tests with JDK11
>
>
> On Thu, Nov 17, 2022 at 11:30 AM Gabor Kaszab <ga...@apache.org>
> wrote:
>
>> Hi Everyone,
>>
>> I propose that we release the following RC as the official Apache Iceberg 1.1.0 release.
>>
>> The commit ID is b3eaf0c6cb9cf6357a925c7443baadd54515a971
>> * This corresponds to the tag: apache-iceberg-1.1.0-rc2
>> * https://github.com/apache/iceberg/commits/apache-iceberg-1.1.0-rc2
>> * https://github.com/apache/iceberg/tree/b3eaf0c6cb9cf6357a925c7443baadd54515a971
>>
>> The release tarball, signature, and checksums are here:
>> * https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-1.1.0-rc2
>>
>> You can find the KEYS file here:
>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>
>> Convenience binary artifacts are staged on Nexus. The Maven repository URL is:
>> * https://repository.apache.org/content/repositories/orgapacheiceberg-1111/
>>
>> Please download, verify, and test.
>>
>> Please vote in the next 72 hours.
>>
>> [ ] +1 Release this as Apache Iceberg 1.1.0
>> [ ] +0
>> [ ] -1 Do not release this because...
>>
>> Again, thanks to *Fokko* for running the RC creation steps for me!
>>
>>
Re: [VOTE] Release Apache Iceberg 1.1.0 RC2
Posted by Eduard Tudenhoefner <ed...@tabular.io>.
+1 (non-binding)
- validated checksum and signature
- checked license docs & ran RAT checks
- ran build and tests with JDK11
On Thu, Nov 17, 2022 at 11:30 AM Gabor Kaszab <ga...@apache.org>
wrote:
> Hi Everyone,
>
> I propose that we release the following RC as the official Apache Iceberg 1.1.0 release.
>
> The commit ID is b3eaf0c6cb9cf6357a925c7443baadd54515a971
> * This corresponds to the tag: apache-iceberg-1.1.0-rc2
> * https://github.com/apache/iceberg/commits/apache-iceberg-1.1.0-rc2
> * https://github.com/apache/iceberg/tree/b3eaf0c6cb9cf6357a925c7443baadd54515a971
>
> The release tarball, signature, and checksums are here:
> * https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-1.1.0-rc2
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>
> Convenience binary artifacts are staged on Nexus. The Maven repository URL is:
> * https://repository.apache.org/content/repositories/orgapacheiceberg-1111/
>
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as Apache Iceberg 1.1.0
> [ ] +0
> [ ] -1 Do not release this because...
>
> Again, thanks to *Fokko* for running the RC creation steps for me!
>
>