You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Jack Ye <ye...@gmail.com> on 2021/01/22 23:26:28 UTC
[VOTE] Release Apache Iceberg 0.11.0 RC0
Hi everyone,
I propose the following RC to be released as the official Apache Iceberg
0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
This corresponds to the tag: apache-iceberg-0.11.0-rc0
* https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
* https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/iceberg/KEYS
Convenience binary artifacts are staged in Nexus. The Maven repository URL
is:
* https://repository.apache.org/content/repositories/orgapacheiceberg-1015
This release includes the following changes:
*High-level features*
- Core API now supports partition spec and sort order evolution
- Spark 3 now supports the following SQL extensions:
- MERGE INTO
- DELETE FROM
- ALTER TABLE ... ADD/DROP PARTITION
- ALTER TABLE ... WRITE ORDERED BY
- invoke stored procedures using CALL
- Flink now supports streaming reads, CDC writes (experimental), and
filter pushdown
- AWS module is added to support better integration with AWS, with AWS
Glue catalog <https://aws.amazon.com/glue> support and dedicated S3
FileIO implementation
- Nessie module is added to support integration with project Nessie
<https://projectnessie.org>
*Important bug fixes*
- #1981 fixes date and timestamp transforms
- #2091 fixes Parquet vectorized reads when column types are promoted
- #1962 fixes Parquet vectorized position reader
- #1991 fixes Avro schema conversions to preserve field docs
- #1811 makes refreshing Spark cache optional
- #1798 fixes read failure when encountering duplicate entries of data
files
- #1785 fixes invalidation of metadata tables in CachingCatalog
- #1784 fixes resolving of SparkSession table's metadata tables
*Other notable changes*
- NaN counter is added to format v2 metrics
- Shared catalog properties are added in core library to standardize
catalog level configurations
- Spark and Flink now supports dynamically loading customized `Catalog`
and `FileIO` implementations
- Spark now supports loading tables with file paths via HadoopTables
- Spark 2 now supports loading tables from other catalogs, like Spark 3
- Spark 3 now supports catalog names in DataFrameReader when using
Iceberg as a format
- Hive now supports INSERT INTO, case insensitive query, projection
pushdown, create DDL with schema and auto type conversion
- ORC now supports reading tinyint, smallint, char, varchar types
- Hadoop catalog now supports role-based access of table listing
Please download, verify, and test.
Please vote in the next 72 hours.
[ ] +1 Release this as Apache Iceberg 0.11.0
[ ] +0
[ ] -1 Do not release this because...
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by Yan Yan <yy...@gmail.com>.
+1 after I did the following verifications and ran a few sample spark
commands:
1. Download the source tarball, signature (.asc), and checksum (.sha512):
OK
2. Import gpg keys: download KEYS and run gpg --import
/path/to/downloaded/KEYS (optional if this hasn’t changed) : OK
3. Verify the signature by running: gpg --verify
apache-iceberg-xx.tar.gz.asc: OK
4. Verify the checksum by running: sha512sum -c
apache-iceberg-xx.tar.gz.sha512 : OK
5. Untar the archive and go into the source directory: tar xzf
apache-iceberg-xx.tar.gz && cd apache-iceberg-xx: OK
6. Run RAT checks to validate license headers: dev/check-license: OK
7. Build and test the project: ./gradlew build (use Java 8) : OK
Not sure if the important bug fixes/other notable changes are just for this
email or part of the release note, if latter than I have some quick
comments: I think #1761 <https://github.com/apache/iceberg/issues/1761>
might be worth mentioning in bug fixes, as it fixed an edge case for ORC
files where evaluators skip including certain files; as well as #1960
<https://github.com/apache/iceberg/pull/1960/> which fixes ORC writers not
collecting metrics per configuration. In other notable changes, I'm not
sure if we need to mention the addition of NaN counters, as it currently
only has very limited support (only in parquet), and the spec change that
mentions it is already merged.
Thanks,
Yan
On Mon, Jan 25, 2021 at 4:46 AM Peter Vary <pv...@cloudera.com.invalid>
wrote:
> +1 (binding) from my side
>
> Here are the checks what I have done:
>
> - Downloaded the source and built the code
> - Checked the size of the iceberg-hive-runtime-0.11.0.jar
> - Tried out the jar on a CDP cluster using Hue
> - Create a table
> - Inserted values into the table
> - Selected the value from the table
> - Selected the value from the table with join
> - Created another table and inserted values into it using a select
>
>
> Here are the settings I have used in my tests:
>
> *SET iceberg.mr.catalog=hive;*
> *SET hive.execution.engine=mr;*
> *ADD JAR /tmp/iceberg-hive-runtime-0.11.0.jar;*
> *ADD JAR /opt/cloudera/parcels/CDH/jars/libfb303-0.9.3.jar*
>
>
> It might be good to know that we have problems writing several types ATM.
> See: https://github.com/apache/iceberg/pull/2126
> I do not think it is blocker since writes are only experimental.
>
> Thanks,
> Peter
>
> On Jan 25, 2021, at 10:24, Ryan Murray <ry...@dremio.com> wrote:
>
> I have moved back to +1 (non-binding)
>
> As you said Ryan, the error message is bad and hides the real error. While
> I was testing misconfigured catalogs I kept getting the error 'Cannot
> initialize Catalog, missing no-arg constructor:
> org.apache.iceberg.hive.HiveCatalog' when the real error is (in this case)
> a misconfigured Hive. I have raised #2145
> <https://github.com/apache/iceberg/issues/2145> to address this as it
> isn't critical to the release.
>
> Best,
> Ryan
>
> On Mon, Jan 25, 2021 at 12:05 AM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> +1 (binding)
>>
>> Downloaded, validated checksum and signature, ran RAT checks, built
>> binaries and tested.
>>
>> Also checked Spark 2, Spark 3, and Hive 2:
>>
>> - Created a new table in Spark 3.1.1 release candidate without the
>> USING clause
>> - Created a table in Spark 3.0.1 with CTAS and a USING clause
>> - Created a new database in Spark 3.0.1 and validated the warehouse
>> location for new tables
>> - Used Spark 3 extensions in 3.0.1 to add bucketing to a table
>> - Deleted data from a table in Spark 3.0.1
>> - Ran merge statements in Spark 3.0.1 and validated join type
>> optimizations
>> - Used multi-catalog support in Spark 2.4.5 to read from testhive and
>> prodhive catalogs using the same config as Spark 3
>> - Tested multi-catalog metadata tables in Spark 2.4.5
>> - Tested input_file_name() in Spark 2.4.5
>> - Read from a Hive catalog table in Hive 2
>>
>> Here’s my command to start Spark 3:
>>
>> /home/blue/Apps/spark-3.0.1-bin-hadoop2.7/bin/spark-shell \
>> --driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 \
>> --conf spark.jars.repositories=https://repository.apache.org/content/repositories/orgapacheiceberg-1015/ \
>> --packages org.apache.iceberg:iceberg-spark3-runtime:0.11.0 \
>> --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
>> --conf spark.hadoop.hive.metastore.uris=thrift://localhost:32917 \
>> --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
>> --conf spark.sql.catalog.local.type=hadoop \
>> --conf spark.sql.catalog.local.warehouse=/home/blue/tmp/hadoop-warehouse \
>> --conf spark.sql.catalog.prodhive=org.apache.iceberg.spark.SparkCatalog \
>> --conf spark.sql.catalog.prodhive.type=hive \
>> --conf spark.sql.catalog.prodhive.warehouse=/home/blue/tmp/prod-warehouse \
>> --conf spark.sql.catalog.prodhive.default-namespace=default \
>> --conf spark.sql.catalog.testhive=org.apache.iceberg.spark.SparkCatalog \
>> --conf spark.sql.catalog.testhive.type=hive \
>> --conf spark.sql.catalog.testhive.uri=thrift://localhost:34847 \
>> --conf spark.sql.catalog.testhive.warehouse=/home/blue/tmp/test-warehouse \
>> --conf spark.sql.catalog.testhive.default-namespace=default \
>> --conf spark.sql.defaultCatalog=prodhive
>>
>> And here’s a script to start Hive:
>>
>> /home/blue/Apps/apache-hive-2.3.7-bin/bin/hive --hiveconf hive.metastore.uris=thrift://localhost:32917
>> hive> SET iceberg.mr.catalog=hive;
>> hive> ADD JAR /home/blue/Downloads/iceberg-hive-runtime-0.11.0.jar;
>>
>> The only issue I found is that the Spark 3.1.1 release candidate can’t
>> use the extensions module because an internal variable substitution class
>> changed in 3.1.x. I don’t think that should fail this release, we can do
>> more thorough testing with 3.1.1 once it is released and fix problems in a
>> point release.
>>
>> On Fri, Jan 22, 2021 at 3:26 PM Jack Ye <ye...@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> I propose the following RC to be released as the official Apache Iceberg
>>> 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>>>
>>> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>>>
>>> This corresponds to the tag: apache-iceberg-0.11.0-rc0
>>> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
>>> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
>>>
>>> The release tarball, signature, and checksums are here:
>>> *
>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
>>>
>>> You can find the KEYS file here:
>>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>>
>>> Convenience binary artifacts are staged in Nexus. The Maven repository
>>> URL is:
>>> *
>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>>>
>>> This release includes the following changes:
>>>
>>> *High-level features*
>>>
>>> - Core API now supports partition spec and sort order evolution
>>> - Spark 3 now supports the following SQL extensions:
>>> - MERGE INTO
>>> - DELETE FROM
>>> - ALTER TABLE ... ADD/DROP PARTITION
>>> - ALTER TABLE ... WRITE ORDERED BY
>>> - invoke stored procedures using CALL
>>> - Flink now supports streaming reads, CDC writes (experimental), and
>>> filter pushdown
>>> - AWS module is added to support better integration with AWS, with AWS
>>> Glue catalog <https://aws.amazon.com/glue> support and dedicated S3
>>> FileIO implementation
>>> - Nessie module is added to support integration with project Nessie
>>> <https://projectnessie.org/>
>>>
>>> *Important bug fixes*
>>>
>>> - #1981 fixes date and timestamp transforms
>>> - #2091 fixes Parquet vectorized reads when column types are promoted
>>> - #1962 fixes Parquet vectorized position reader
>>> - #1991 fixes Avro schema conversions to preserve field docs
>>> - #1811 makes refreshing Spark cache optional
>>> - #1798 fixes read failure when encountering duplicate entries of
>>> data files
>>> - #1785 fixes invalidation of metadata tables in CachingCatalog
>>> - #1784 fixes resolving of SparkSession table's metadata tables
>>>
>>> *Other notable changes*
>>>
>>> - NaN counter is added to format v2 metrics
>>> - Shared catalog properties are added in core library to standardize
>>> catalog level configurations
>>> - Spark and Flink now supports dynamically loading customized
>>> `Catalog` and `FileIO` implementations
>>> - Spark now supports loading tables with file paths via HadoopTables
>>> - Spark 2 now supports loading tables from other catalogs, like
>>> Spark 3
>>> - Spark 3 now supports catalog names in DataFrameReader when using
>>> Iceberg as a format
>>> - Hive now supports INSERT INTO, case insensitive query, projection
>>> pushdown, create DDL with schema and auto type conversion
>>> - ORC now supports reading tinyint, smallint, char, varchar types
>>> - Hadoop catalog now supports role-based access of table listing
>>>
>>> Please download, verify, and test.
>>>
>>> Please vote in the next 72 hours.
>>>
>>> [ ] +1 Release this as Apache Iceberg 0.11.0
>>> [ ] +0
>>> [ ] -1 Do not release this because...
>>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
>
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by Peter Vary <pv...@cloudera.com.INVALID>.
+1 (binding) from my side
Here are the checks what I have done:
Downloaded the source and built the code
Checked the size of the iceberg-hive-runtime-0.11.0.jar
Tried out the jar on a CDP cluster using Hue
Create a table
Inserted values into the table
Selected the value from the table
Selected the value from the table with join
Created another table and inserted values into it using a select
Here are the settings I have used in my tests:
SET iceberg.mr.catalog=hive;
SET hive.execution.engine=mr;
ADD JAR /tmp/iceberg-hive-runtime-0.11.0.jar;
ADD JAR /opt/cloudera/parcels/CDH/jars/libfb303-0.9.3.jar
It might be good to know that we have problems writing several types ATM. See: https://github.com/apache/iceberg/pull/2126 <https://github.com/apache/iceberg/pull/2126>
I do not think it is blocker since writes are only experimental.
Thanks,
Peter
> On Jan 25, 2021, at 10:24, Ryan Murray <ry...@dremio.com> wrote:
>
> I have moved back to +1 (non-binding)
>
> As you said Ryan, the error message is bad and hides the real error. While I was testing misconfigured catalogs I kept getting the error 'Cannot initialize Catalog, missing no-arg constructor: org.apache.iceberg.hive.HiveCatalog' when the real error is (in this case) a misconfigured Hive. I have raised #2145 <https://github.com/apache/iceberg/issues/2145> to address this as it isn't critical to the release.
>
> Best,
> Ryan
>
> On Mon, Jan 25, 2021 at 12:05 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
> +1 (binding)
>
> Downloaded, validated checksum and signature, ran RAT checks, built binaries and tested.
>
> Also checked Spark 2, Spark 3, and Hive 2:
>
> Created a new table in Spark 3.1.1 release candidate without the USING clause
> Created a table in Spark 3.0.1 with CTAS and a USING clause
> Created a new database in Spark 3.0.1 and validated the warehouse location for new tables
> Used Spark 3 extensions in 3.0.1 to add bucketing to a table
> Deleted data from a table in Spark 3.0.1
> Ran merge statements in Spark 3.0.1 and validated join type optimizations
> Used multi-catalog support in Spark 2.4.5 to read from testhive and prodhive catalogs using the same config as Spark 3
> Tested multi-catalog metadata tables in Spark 2.4.5
> Tested input_file_name() in Spark 2.4.5
> Read from a Hive catalog table in Hive 2
> Here’s my command to start Spark 3:
>
> /home/blue/Apps/spark-3.0.1-bin-hadoop2.7/bin/spark-shell \
> --driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 \
> --conf spark.jars.repositories=https://repository.apache.org/content/repositories/orgapacheiceberg-1015/ <https://repository.apache.org/content/repositories/orgapacheiceberg-1015/> \
> --packages org.apache.iceberg:iceberg-spark3-runtime:0.11.0 \
> --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
> --conf spark.hadoop.hive.metastore.uris=thrift://localhost:32917 \
> --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
> --conf spark.sql.catalog.local.type=hadoop \
> --conf spark.sql.catalog.local.warehouse=/home/blue/tmp/hadoop-warehouse \
> --conf spark.sql.catalog.prodhive=org.apache.iceberg.spark.SparkCatalog \
> --conf spark.sql.catalog.prodhive.type=hive \
> --conf spark.sql.catalog.prodhive.warehouse=/home/blue/tmp/prod-warehouse \
> --conf spark.sql.catalog.prodhive.default-namespace=default \
> --conf spark.sql.catalog.testhive=org.apache.iceberg.spark.SparkCatalog \
> --conf spark.sql.catalog.testhive.type=hive \
> --conf spark.sql.catalog.testhive.uri=thrift://localhost:34847 \
> --conf spark.sql.catalog.testhive.warehouse=/home/blue/tmp/test-warehouse \
> --conf spark.sql.catalog.testhive.default-namespace=default \
> --conf spark.sql.defaultCatalog=prodhive
> And here’s a script to start Hive:
>
> /home/blue/Apps/apache-hive-2.3.7-bin/bin/hive --hiveconf hive.metastore.uris=thrift://localhost:32917
> hive> SET iceberg.mr.catalog=hive;
> hive> ADD JAR /home/blue/Downloads/iceberg-hive-runtime-0.11.0.jar;
> The only issue I found is that the Spark 3.1.1 release candidate can’t use the extensions module because an internal variable substitution class changed in 3.1.x. I don’t think that should fail this release, we can do more thorough testing with 3.1.1 once it is released and fix problems in a point release.
>
>
> On Fri, Jan 22, 2021 at 3:26 PM Jack Ye <yezhaoqin@gmail.com <ma...@gmail.com>> wrote:
> Hi everyone,
>
> I propose the following RC to be released as the official Apache Iceberg 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>
> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>
> This corresponds to the tag: apache-iceberg-0.11.0-rc0
> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0 <https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0>
> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0 <https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0>
>
> The release tarball, signature, and checksums are here:
> * https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0 <https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0>
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS <https://dist.apache.org/repos/dist/dev/iceberg/KEYS>
>
> Convenience binary artifacts are staged in Nexus. The Maven repository URL is:
> * https://repository.apache.org/content/repositories/orgapacheiceberg-1015 <https://repository.apache.org/content/repositories/orgapacheiceberg-1015>
>
> This release includes the following changes:
>
> High-level features
> Core API now supports partition spec and sort order evolution
> Spark 3 now supports the following SQL extensions:
> MERGE INTO
> DELETE FROM
> ALTER TABLE ... ADD/DROP PARTITION
> ALTER TABLE ... WRITE ORDERED BY
> invoke stored procedures using CALL
> Flink now supports streaming reads, CDC writes (experimental), and filter pushdown
> AWS module is added to support better integration with AWS, with AWS Glue catalog <https://aws.amazon.com/glue> support and dedicated S3 FileIO implementation
> Nessie module is added to support integration with project Nessie <https://projectnessie.org/>
> Important bug fixes
> #1981 fixes date and timestamp transforms
> #2091 fixes Parquet vectorized reads when column types are promoted
> #1962 fixes Parquet vectorized position reader
> #1991 fixes Avro schema conversions to preserve field docs
> #1811 makes refreshing Spark cache optional
> #1798 fixes read failure when encountering duplicate entries of data files
> #1785 fixes invalidation of metadata tables in CachingCatalog
> #1784 fixes resolving of SparkSession table's metadata tables
> Other notable changes
> NaN counter is added to format v2 metrics
> Shared catalog properties are added in core library to standardize catalog level configurations
> Spark and Flink now supports dynamically loading customized `Catalog` and `FileIO` implementations
> Spark now supports loading tables with file paths via HadoopTables
> Spark 2 now supports loading tables from other catalogs, like Spark 3
> Spark 3 now supports catalog names in DataFrameReader when using Iceberg as a format
> Hive now supports INSERT INTO, case insensitive query, projection pushdown, create DDL with schema and auto type conversion
> ORC now supports reading tinyint, smallint, char, varchar types
> Hadoop catalog now supports role-based access of table listing
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as Apache Iceberg 0.11.0
> [ ] +0
> [ ] -1 Do not release this because...
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by Ryan Murray <ry...@dremio.com>.
I have moved back to +1 (non-binding)
As you said Ryan, the error message is bad and hides the real error. While
I was testing misconfigured catalogs I kept getting the error 'Cannot
initialize Catalog, missing no-arg constructor:
org.apache.iceberg.hive.HiveCatalog' when the real error is (in this case)
a misconfigured Hive. I have raised #2145
<https://github.com/apache/iceberg/issues/2145> to address this as it isn't
critical to the release.
Best,
Ryan
On Mon, Jan 25, 2021 at 12:05 AM Ryan Blue <rb...@netflix.com.invalid>
wrote:
> +1 (binding)
>
> Downloaded, validated checksum and signature, ran RAT checks, built
> binaries and tested.
>
> Also checked Spark 2, Spark 3, and Hive 2:
>
> - Created a new table in Spark 3.1.1 release candidate without the
> USING clause
> - Created a table in Spark 3.0.1 with CTAS and a USING clause
> - Created a new database in Spark 3.0.1 and validated the warehouse
> location for new tables
> - Used Spark 3 extensions in 3.0.1 to add bucketing to a table
> - Deleted data from a table in Spark 3.0.1
> - Ran merge statements in Spark 3.0.1 and validated join type
> optimizations
> - Used multi-catalog support in Spark 2.4.5 to read from testhive and
> prodhive catalogs using the same config as Spark 3
> - Tested multi-catalog metadata tables in Spark 2.4.5
> - Tested input_file_name() in Spark 2.4.5
> - Read from a Hive catalog table in Hive 2
>
> Here’s my command to start Spark 3:
>
> /home/blue/Apps/spark-3.0.1-bin-hadoop2.7/bin/spark-shell \
> --driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 \
> --conf spark.jars.repositories=https://repository.apache.org/content/repositories/orgapacheiceberg-1015/ \
> --packages org.apache.iceberg:iceberg-spark3-runtime:0.11.0 \
> --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
> --conf spark.hadoop.hive.metastore.uris=thrift://localhost:32917 \
> --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
> --conf spark.sql.catalog.local.type=hadoop \
> --conf spark.sql.catalog.local.warehouse=/home/blue/tmp/hadoop-warehouse \
> --conf spark.sql.catalog.prodhive=org.apache.iceberg.spark.SparkCatalog \
> --conf spark.sql.catalog.prodhive.type=hive \
> --conf spark.sql.catalog.prodhive.warehouse=/home/blue/tmp/prod-warehouse \
> --conf spark.sql.catalog.prodhive.default-namespace=default \
> --conf spark.sql.catalog.testhive=org.apache.iceberg.spark.SparkCatalog \
> --conf spark.sql.catalog.testhive.type=hive \
> --conf spark.sql.catalog.testhive.uri=thrift://localhost:34847 \
> --conf spark.sql.catalog.testhive.warehouse=/home/blue/tmp/test-warehouse \
> --conf spark.sql.catalog.testhive.default-namespace=default \
> --conf spark.sql.defaultCatalog=prodhive
>
> And here’s a script to start Hive:
>
> /home/blue/Apps/apache-hive-2.3.7-bin/bin/hive --hiveconf hive.metastore.uris=thrift://localhost:32917
> hive> SET iceberg.mr.catalog=hive;
> hive> ADD JAR /home/blue/Downloads/iceberg-hive-runtime-0.11.0.jar;
>
> The only issue I found is that the Spark 3.1.1 release candidate can’t use
> the extensions module because an internal variable substitution class
> changed in 3.1.x. I don’t think that should fail this release, we can do
> more thorough testing with 3.1.1 once it is released and fix problems in a
> point release.
>
> On Fri, Jan 22, 2021 at 3:26 PM Jack Ye <ye...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> I propose the following RC to be released as the official Apache Iceberg
>> 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>>
>> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>>
>> This corresponds to the tag: apache-iceberg-0.11.0-rc0
>> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
>> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
>>
>> The release tarball, signature, and checksums are here:
>> *
>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
>>
>> You can find the KEYS file here:
>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>
>> Convenience binary artifacts are staged in Nexus. The Maven repository
>> URL is:
>> *
>> https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>>
>> This release includes the following changes:
>>
>> *High-level features*
>>
>> - Core API now supports partition spec and sort order evolution
>> - Spark 3 now supports the following SQL extensions:
>> - MERGE INTO
>> - DELETE FROM
>> - ALTER TABLE ... ADD/DROP PARTITION
>> - ALTER TABLE ... WRITE ORDERED BY
>> - invoke stored procedures using CALL
>> - Flink now supports streaming reads, CDC writes (experimental), and
>> filter pushdown
>> - AWS module is added to support better integration with AWS, with AWS
>> Glue catalog <https://aws.amazon.com/glue> support and dedicated S3
>> FileIO implementation
>> - Nessie module is added to support integration with project Nessie
>> <https://projectnessie.org>
>>
>> *Important bug fixes*
>>
>> - #1981 fixes date and timestamp transforms
>> - #2091 fixes Parquet vectorized reads when column types are promoted
>> - #1962 fixes Parquet vectorized position reader
>> - #1991 fixes Avro schema conversions to preserve field docs
>> - #1811 makes refreshing Spark cache optional
>> - #1798 fixes read failure when encountering duplicate entries of
>> data files
>> - #1785 fixes invalidation of metadata tables in CachingCatalog
>> - #1784 fixes resolving of SparkSession table's metadata tables
>>
>> *Other notable changes*
>>
>> - NaN counter is added to format v2 metrics
>> - Shared catalog properties are added in core library to standardize
>> catalog level configurations
>> - Spark and Flink now supports dynamically loading customized
>> `Catalog` and `FileIO` implementations
>> - Spark now supports loading tables with file paths via HadoopTables
>> - Spark 2 now supports loading tables from other catalogs, like Spark
>> 3
>> - Spark 3 now supports catalog names in DataFrameReader when using
>> Iceberg as a format
>> - Hive now supports INSERT INTO, case insensitive query, projection
>> pushdown, create DDL with schema and auto type conversion
>> - ORC now supports reading tinyint, smallint, char, varchar types
>> - Hadoop catalog now supports role-based access of table listing
>>
>> Please download, verify, and test.
>>
>> Please vote in the next 72 hours.
>>
>> [ ] +1 Release this as Apache Iceberg 0.11.0
>> [ ] +0
>> [ ] -1 Do not release this because...
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
+1 (binding)
Downloaded, validated checksum and signature, ran RAT checks, built
binaries and tested.
Also checked Spark 2, Spark 3, and Hive 2:
- Created a new table in Spark 3.1.1 release candidate without the USING
clause
- Created a table in Spark 3.0.1 with CTAS and a USING clause
- Created a new database in Spark 3.0.1 and validated the warehouse
location for new tables
- Used Spark 3 extensions in 3.0.1 to add bucketing to a table
- Deleted data from a table in Spark 3.0.1
- Ran merge statements in Spark 3.0.1 and validated join type
optimizations
- Used multi-catalog support in Spark 2.4.5 to read from testhive and
prodhive catalogs using the same config as Spark 3
- Tested multi-catalog metadata tables in Spark 2.4.5
- Tested input_file_name() in Spark 2.4.5
- Read from a Hive catalog table in Hive 2
Here’s my command to start Spark 3:
/home/blue/Apps/spark-3.0.1-bin-hadoop2.7/bin/spark-shell \
--driver-java-options
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 \
--conf spark.jars.repositories=https://repository.apache.org/content/repositories/orgapacheiceberg-1015/
\
--packages org.apache.iceberg:iceberg-spark3-runtime:0.11.0 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
--conf spark.hadoop.hive.metastore.uris=thrift://localhost:32917 \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.local.type=hadoop \
--conf spark.sql.catalog.local.warehouse=/home/blue/tmp/hadoop-warehouse \
--conf spark.sql.catalog.prodhive=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.prodhive.type=hive \
--conf spark.sql.catalog.prodhive.warehouse=/home/blue/tmp/prod-warehouse \
--conf spark.sql.catalog.prodhive.default-namespace=default \
--conf spark.sql.catalog.testhive=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.testhive.type=hive \
--conf spark.sql.catalog.testhive.uri=thrift://localhost:34847 \
--conf spark.sql.catalog.testhive.warehouse=/home/blue/tmp/test-warehouse \
--conf spark.sql.catalog.testhive.default-namespace=default \
--conf spark.sql.defaultCatalog=prodhive
And here’s a script to start Hive:
/home/blue/Apps/apache-hive-2.3.7-bin/bin/hive --hiveconf
hive.metastore.uris=thrift://localhost:32917
hive> SET iceberg.mr.catalog=hive;
hive> ADD JAR /home/blue/Downloads/iceberg-hive-runtime-0.11.0.jar;
The only issue I found is that the Spark 3.1.1 release candidate can’t use
the extensions module because an internal variable substitution class
changed in 3.1.x. I don’t think that should fail this release, we can do
more thorough testing with 3.1.1 once it is released and fix problems in a
point release.
On Fri, Jan 22, 2021 at 3:26 PM Jack Ye <ye...@gmail.com> wrote:
> Hi everyone,
>
> I propose the following RC to be released as the official Apache Iceberg
> 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>
> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>
> This corresponds to the tag: apache-iceberg-0.11.0-rc0
> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
>
> The release tarball, signature, and checksums are here:
> * https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>
> Convenience binary artifacts are staged in Nexus. The Maven repository URL
> is:
> * https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>
> This release includes the following changes:
>
> *High-level features*
>
> - Core API now supports partition spec and sort order evolution
> - Spark 3 now supports the following SQL extensions:
> - MERGE INTO
> - DELETE FROM
> - ALTER TABLE ... ADD/DROP PARTITION
> - ALTER TABLE ... WRITE ORDERED BY
> - invoke stored procedures using CALL
> - Flink now supports streaming reads, CDC writes (experimental), and
> filter pushdown
> - AWS module is added to support better integration with AWS, with AWS
> Glue catalog <https://aws.amazon.com/glue> support and dedicated S3
> FileIO implementation
> - Nessie module is added to support integration with project Nessie
> <https://projectnessie.org>
>
> *Important bug fixes*
>
> - #1981 fixes date and timestamp transforms
> - #2091 fixes Parquet vectorized reads when column types are promoted
> - #1962 fixes Parquet vectorized position reader
> - #1991 fixes Avro schema conversions to preserve field docs
> - #1811 makes refreshing Spark cache optional
> - #1798 fixes read failure when encountering duplicate entries of data
> files
> - #1785 fixes invalidation of metadata tables in CachingCatalog
> - #1784 fixes resolving of SparkSession table's metadata tables
>
> *Other notable changes*
>
> - NaN counter is added to format v2 metrics
> - Shared catalog properties are added in core library to standardize
> catalog level configurations
> - Spark and Flink now supports dynamically loading customized
> `Catalog` and `FileIO` implementations
> - Spark now supports loading tables with file paths via HadoopTables
> - Spark 2 now supports loading tables from other catalogs, like Spark 3
> - Spark 3 now supports catalog names in DataFrameReader when using
> Iceberg as a format
> - Hive now supports INSERT INTO, case insensitive query, projection
> pushdown, create DDL with schema and auto type conversion
> - ORC now supports reading tinyint, smallint, char, varchar types
> - Hadoop catalog now supports role-based access of table listing
>
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as Apache Iceberg 0.11.0
> [ ] +0
> [ ] -1 Do not release this because...
>
--
Ryan Blue
Software Engineer
Netflix
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by John Zhuge <jz...@apache.org>.
Ignore my last email. Vote already passed.
On Tue, Jan 26, 2021 at 11:40 AM John Zhuge <jz...@apache.org> wrote:
> Got 404 for these 2 links:
>
> -
> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
> -
> https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>
> Anybody saw the same issue?
>
> On Tue, Jan 26, 2021 at 10:37 AM Jack Ye <ye...@gmail.com> wrote:
>
>> +1, binding.
>> Verified the signature, license, checksum and tests. Also ran AWS
>> integration tests and all passed.
>> -Jack
>>
>> On Tue, Jan 26, 2021 at 9:56 AM Daniel Weeks <dw...@apache.org> wrote:
>>
>>> +1 (binding)
>>>
>>> Verified signatures, checksums, license, build and tests.
>>>
>>> On Tue, Jan 26, 2021 at 9:11 AM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>>> OpenInx, we can update documentation any time. It should not block the
>>>> release, just open a PR and we'll merge it and push docs after that.
>>>>
>>>> On Mon, Jan 25, 2021 at 9:01 PM OpenInx <op...@gmail.com> wrote:
>>>>
>>>>> Hi dev
>>>>>
>>>>> I'd like to include this patch in release 0.11.0 because it's the
>>>>> document of new flink features. I'm sorry that I did not update the
>>>>> flink's document in time when the feature code merged, but I think it's
>>>>> worth it to merge this document PR when we release iceberg 0.11.0, that
>>>>> helps a lot for users who want to use those new features, such as streaming
>>>>> reader, rewrite data files action, write distribution to cluster data.
>>>>> (I will update those documents in time so that we won't rollback the next
>>>>> release).
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> On Tue, Jan 26, 2021 at 10:17 AM Anton Okolnychyi
>>>>> <ao...@apple.com.invalid> wrote:
>>>>>
>>>>>> +1 (binding)
>>>>>>
>>>>>> I did local tests with Spark 3.0.1. I think we should also note the
>>>>>> support for DELETE FROM and MERGE INTO in Spark is experimental.
>>>>>>
>>>>>> Thanks,
>>>>>> Anton
>>>>>>
>>>>>> On 22 Jan 2021, at 15:26, Jack Ye <ye...@gmail.com> wrote:
>>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> I propose the following RC to be released as the official Apache
>>>>>> Iceberg 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>>>>>>
>>>>>> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>>>>>>
>>>>>> This corresponds to the tag: apache-iceberg-0.11.0-rc0
>>>>>> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
>>>>>> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
>>>>>>
>>>>>> The release tarball, signature, and checksums are here:
>>>>>> *
>>>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
>>>>>>
>>>>>> You can find the KEYS file here:
>>>>>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>>>>>
>>>>>> Convenience binary artifacts are staged in Nexus. The Maven
>>>>>> repository URL is:
>>>>>> *
>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>>>>>>
>>>>>> This release includes the following changes:
>>>>>>
>>>>>> *High-level features*
>>>>>>
>>>>>> - Core API now supports partition spec and sort order evolution
>>>>>> - Spark 3 now supports the following SQL extensions:
>>>>>> - MERGE INTO
>>>>>> - DELETE FROM
>>>>>> - ALTER TABLE ... ADD/DROP PARTITION
>>>>>> - ALTER TABLE ... WRITE ORDERED BY
>>>>>> - invoke stored procedures using CALL
>>>>>> - Flink now supports streaming reads, CDC writes (experimental),
>>>>>> and filter pushdown
>>>>>> - AWS module is added to support better integration with AWS,
>>>>>> with AWS Glue catalog <https://aws.amazon.com/glue> support and
>>>>>> dedicated S3 FileIO implementation
>>>>>> - Nessie module is added to support integration with project
>>>>>> Nessie <https://projectnessie.org/>
>>>>>>
>>>>>> *Important bug fixes*
>>>>>>
>>>>>> - #1981 fixes date and timestamp transforms
>>>>>> - #2091 fixes Parquet vectorized reads when column types are
>>>>>> promoted
>>>>>> - #1962 fixes Parquet vectorized position reader
>>>>>> - #1991 fixes Avro schema conversions to preserve field docs
>>>>>> - #1811 makes refreshing Spark cache optional
>>>>>> - #1798 fixes read failure when encountering duplicate entries of
>>>>>> data files
>>>>>> - #1785 fixes invalidation of metadata tables in CachingCatalog
>>>>>> - #1784 fixes resolving of SparkSession table's metadata tables
>>>>>>
>>>>>> *Other notable changes*
>>>>>>
>>>>>> - NaN counter is added to format v2 metrics
>>>>>> - Shared catalog properties are added in core library to
>>>>>> standardize catalog level configurations
>>>>>> - Spark and Flink now supports dynamically loading customized
>>>>>> `Catalog` and `FileIO` implementations
>>>>>> - Spark now supports loading tables with file paths via
>>>>>> HadoopTables
>>>>>> - Spark 2 now supports loading tables from other catalogs, like
>>>>>> Spark 3
>>>>>> - Spark 3 now supports catalog names in DataFrameReader when
>>>>>> using Iceberg as a format
>>>>>> - Hive now supports INSERT INTO, case insensitive query,
>>>>>> projection pushdown, create DDL with schema and auto type conversion
>>>>>> - ORC now supports reading tinyint, smallint, char, varchar types
>>>>>> - Hadoop catalog now supports role-based access of table listing
>>>>>>
>>>>>> Please download, verify, and test.
>>>>>>
>>>>>> Please vote in the next 72 hours.
>>>>>>
>>>>>> [ ] +1 Release this as Apache Iceberg 0.11.0
>>>>>> [ ] +0
>>>>>> [ ] -1 Do not release this because...
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
>
> --
> John Zhuge
>
--
John Zhuge
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by John Zhuge <jz...@apache.org>.
Got 404 for these 2 links:
-
https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
-
https://repository.apache.org/content/repositories/orgapacheiceberg-1015
Anybody saw the same issue?
On Tue, Jan 26, 2021 at 10:37 AM Jack Ye <ye...@gmail.com> wrote:
> +1, binding.
> Verified the signature, license, checksum and tests. Also ran AWS
> integration tests and all passed.
> -Jack
>
> On Tue, Jan 26, 2021 at 9:56 AM Daniel Weeks <dw...@apache.org> wrote:
>
>> +1 (binding)
>>
>> Verified signatures, checksums, license, build and tests.
>>
>> On Tue, Jan 26, 2021 at 9:11 AM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>>> OpenInx, we can update documentation any time. It should not block the
>>> release, just open a PR and we'll merge it and push docs after that.
>>>
>>> On Mon, Jan 25, 2021 at 9:01 PM OpenInx <op...@gmail.com> wrote:
>>>
>>>> Hi dev
>>>>
>>>> I'd like to include this patch in release 0.11.0 because it's the
>>>> document of new flink features. I'm sorry that I did not update the
>>>> flink's document in time when the feature code merged, but I think it's
>>>> worth it to merge this document PR when we release iceberg 0.11.0, that
>>>> helps a lot for users who want to use those new features, such as streaming
>>>> reader, rewrite data files action, write distribution to cluster data.
>>>> (I will update those documents in time so that we won't rollback the next
>>>> release).
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> On Tue, Jan 26, 2021 at 10:17 AM Anton Okolnychyi
>>>> <ao...@apple.com.invalid> wrote:
>>>>
>>>>> +1 (binding)
>>>>>
>>>>> I did local tests with Spark 3.0.1. I think we should also note the
>>>>> support for DELETE FROM and MERGE INTO in Spark is experimental.
>>>>>
>>>>> Thanks,
>>>>> Anton
>>>>>
>>>>> On 22 Jan 2021, at 15:26, Jack Ye <ye...@gmail.com> wrote:
>>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I propose the following RC to be released as the official Apache
>>>>> Iceberg 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>>>>>
>>>>> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>>>>>
>>>>> This corresponds to the tag: apache-iceberg-0.11.0-rc0
>>>>> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
>>>>> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
>>>>>
>>>>> The release tarball, signature, and checksums are here:
>>>>> *
>>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
>>>>>
>>>>> You can find the KEYS file here:
>>>>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>>>>
>>>>> Convenience binary artifacts are staged in Nexus. The Maven repository
>>>>> URL is:
>>>>> *
>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>>>>>
>>>>> This release includes the following changes:
>>>>>
>>>>> *High-level features*
>>>>>
>>>>> - Core API now supports partition spec and sort order evolution
>>>>> - Spark 3 now supports the following SQL extensions:
>>>>> - MERGE INTO
>>>>> - DELETE FROM
>>>>> - ALTER TABLE ... ADD/DROP PARTITION
>>>>> - ALTER TABLE ... WRITE ORDERED BY
>>>>> - invoke stored procedures using CALL
>>>>> - Flink now supports streaming reads, CDC writes (experimental),
>>>>> and filter pushdown
>>>>> - AWS module is added to support better integration with AWS, with AWS
>>>>> Glue catalog <https://aws.amazon.com/glue> support and dedicated
>>>>> S3 FileIO implementation
>>>>> - Nessie module is added to support integration with project Nessie
>>>>> <https://projectnessie.org/>
>>>>>
>>>>> *Important bug fixes*
>>>>>
>>>>> - #1981 fixes date and timestamp transforms
>>>>> - #2091 fixes Parquet vectorized reads when column types are
>>>>> promoted
>>>>> - #1962 fixes Parquet vectorized position reader
>>>>> - #1991 fixes Avro schema conversions to preserve field docs
>>>>> - #1811 makes refreshing Spark cache optional
>>>>> - #1798 fixes read failure when encountering duplicate entries of
>>>>> data files
>>>>> - #1785 fixes invalidation of metadata tables in CachingCatalog
>>>>> - #1784 fixes resolving of SparkSession table's metadata tables
>>>>>
>>>>> *Other notable changes*
>>>>>
>>>>> - NaN counter is added to format v2 metrics
>>>>> - Shared catalog properties are added in core library to
>>>>> standardize catalog level configurations
>>>>> - Spark and Flink now supports dynamically loading customized
>>>>> `Catalog` and `FileIO` implementations
>>>>> - Spark now supports loading tables with file paths via
>>>>> HadoopTables
>>>>> - Spark 2 now supports loading tables from other catalogs, like
>>>>> Spark 3
>>>>> - Spark 3 now supports catalog names in DataFrameReader when using
>>>>> Iceberg as a format
>>>>> - Hive now supports INSERT INTO, case insensitive query,
>>>>> projection pushdown, create DDL with schema and auto type conversion
>>>>> - ORC now supports reading tinyint, smallint, char, varchar types
>>>>> - Hadoop catalog now supports role-based access of table listing
>>>>>
>>>>> Please download, verify, and test.
>>>>>
>>>>> Please vote in the next 72 hours.
>>>>>
>>>>> [ ] +1 Release this as Apache Iceberg 0.11.0
>>>>> [ ] +0
>>>>> [ ] -1 Do not release this because...
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
--
John Zhuge
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by Jack Ye <ye...@gmail.com>.
+1, binding.
Verified the signature, license, checksum and tests. Also ran AWS
integration tests and all passed.
-Jack
On Tue, Jan 26, 2021 at 9:56 AM Daniel Weeks <dw...@apache.org> wrote:
> +1 (binding)
>
> Verified signatures, checksums, license, build and tests.
>
> On Tue, Jan 26, 2021 at 9:11 AM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> OpenInx, we can update documentation any time. It should not block the
>> release, just open a PR and we'll merge it and push docs after that.
>>
>> On Mon, Jan 25, 2021 at 9:01 PM OpenInx <op...@gmail.com> wrote:
>>
>>> Hi dev
>>>
>>> I'd like to include this patch in release 0.11.0 because it's the
>>> document of new flink features. I'm sorry that I did not update the
>>> flink's document in time when the feature code merged, but I think it's
>>> worth it to merge this document PR when we release iceberg 0.11.0, that
>>> helps a lot for users who want to use those new features, such as streaming
>>> reader, rewrite data files action, write distribution to cluster data.
>>> (I will update those documents in time so that we won't rollback the next
>>> release).
>>>
>>> Thanks.
>>>
>>>
>>> On Tue, Jan 26, 2021 at 10:17 AM Anton Okolnychyi
>>> <ao...@apple.com.invalid> wrote:
>>>
>>>> +1 (binding)
>>>>
>>>> I did local tests with Spark 3.0.1. I think we should also note the
>>>> support for DELETE FROM and MERGE INTO in Spark is experimental.
>>>>
>>>> Thanks,
>>>> Anton
>>>>
>>>> On 22 Jan 2021, at 15:26, Jack Ye <ye...@gmail.com> wrote:
>>>>
>>>> Hi everyone,
>>>>
>>>> I propose the following RC to be released as the official Apache
>>>> Iceberg 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>>>>
>>>> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>>>>
>>>> This corresponds to the tag: apache-iceberg-0.11.0-rc0
>>>> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
>>>> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
>>>>
>>>> The release tarball, signature, and checksums are here:
>>>> *
>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
>>>>
>>>> You can find the KEYS file here:
>>>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>>>
>>>> Convenience binary artifacts are staged in Nexus. The Maven repository
>>>> URL is:
>>>> *
>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>>>>
>>>> This release includes the following changes:
>>>>
>>>> *High-level features*
>>>>
>>>> - Core API now supports partition spec and sort order evolution
>>>> - Spark 3 now supports the following SQL extensions:
>>>> - MERGE INTO
>>>> - DELETE FROM
>>>> - ALTER TABLE ... ADD/DROP PARTITION
>>>> - ALTER TABLE ... WRITE ORDERED BY
>>>> - invoke stored procedures using CALL
>>>> - Flink now supports streaming reads, CDC writes (experimental),
>>>> and filter pushdown
>>>> - AWS module is added to support better integration with AWS, with AWS
>>>> Glue catalog <https://aws.amazon.com/glue> support and dedicated S3
>>>> FileIO implementation
>>>> - Nessie module is added to support integration with project Nessie
>>>> <https://projectnessie.org/>
>>>>
>>>> *Important bug fixes*
>>>>
>>>> - #1981 fixes date and timestamp transforms
>>>> - #2091 fixes Parquet vectorized reads when column types are
>>>> promoted
>>>> - #1962 fixes Parquet vectorized position reader
>>>> - #1991 fixes Avro schema conversions to preserve field docs
>>>> - #1811 makes refreshing Spark cache optional
>>>> - #1798 fixes read failure when encountering duplicate entries of
>>>> data files
>>>> - #1785 fixes invalidation of metadata tables in CachingCatalog
>>>> - #1784 fixes resolving of SparkSession table's metadata tables
>>>>
>>>> *Other notable changes*
>>>>
>>>> - NaN counter is added to format v2 metrics
>>>> - Shared catalog properties are added in core library to
>>>> standardize catalog level configurations
>>>> - Spark and Flink now supports dynamically loading customized
>>>> `Catalog` and `FileIO` implementations
>>>> - Spark now supports loading tables with file paths via HadoopTables
>>>> - Spark 2 now supports loading tables from other catalogs, like
>>>> Spark 3
>>>> - Spark 3 now supports catalog names in DataFrameReader when using
>>>> Iceberg as a format
>>>> - Hive now supports INSERT INTO, case insensitive query, projection
>>>> pushdown, create DDL with schema and auto type conversion
>>>> - ORC now supports reading tinyint, smallint, char, varchar types
>>>> - Hadoop catalog now supports role-based access of table listing
>>>>
>>>> Please download, verify, and test.
>>>>
>>>> Please vote in the next 72 hours.
>>>>
>>>> [ ] +1 Release this as Apache Iceberg 0.11.0
>>>> [ ] +0
>>>> [ ] -1 Do not release this because...
>>>>
>>>>
>>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by Daniel Weeks <dw...@apache.org>.
+1 (binding)
Verified signatures, checksums, license, build and tests.
On Tue, Jan 26, 2021 at 9:11 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
> OpenInx, we can update documentation any time. It should not block the
> release, just open a PR and we'll merge it and push docs after that.
>
> On Mon, Jan 25, 2021 at 9:01 PM OpenInx <op...@gmail.com> wrote:
>
>> Hi dev
>>
>> I'd like to include this patch in release 0.11.0 because it's the
>> document of new flink features. I'm sorry that I did not update the
>> flink's document in time when the feature code merged, but I think it's
>> worth it to merge this document PR when we release iceberg 0.11.0, that
>> helps a lot for users who want to use those new features, such as streaming
>> reader, rewrite data files action, write distribution to cluster data.
>> (I will update those documents in time so that we won't rollback the next
>> release).
>>
>> Thanks.
>>
>>
>> On Tue, Jan 26, 2021 at 10:17 AM Anton Okolnychyi
>> <ao...@apple.com.invalid> wrote:
>>
>>> +1 (binding)
>>>
>>> I did local tests with Spark 3.0.1. I think we should also note the
>>> support for DELETE FROM and MERGE INTO in Spark is experimental.
>>>
>>> Thanks,
>>> Anton
>>>
>>> On 22 Jan 2021, at 15:26, Jack Ye <ye...@gmail.com> wrote:
>>>
>>> Hi everyone,
>>>
>>> I propose the following RC to be released as the official Apache Iceberg
>>> 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>>>
>>> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>>>
>>> This corresponds to the tag: apache-iceberg-0.11.0-rc0
>>> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
>>> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
>>>
>>> The release tarball, signature, and checksums are here:
>>> *
>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
>>>
>>> You can find the KEYS file here:
>>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>>
>>> Convenience binary artifacts are staged in Nexus. The Maven repository
>>> URL is:
>>> *
>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>>>
>>> This release includes the following changes:
>>>
>>> *High-level features*
>>>
>>> - Core API now supports partition spec and sort order evolution
>>> - Spark 3 now supports the following SQL extensions:
>>> - MERGE INTO
>>> - DELETE FROM
>>> - ALTER TABLE ... ADD/DROP PARTITION
>>> - ALTER TABLE ... WRITE ORDERED BY
>>> - invoke stored procedures using CALL
>>> - Flink now supports streaming reads, CDC writes (experimental), and
>>> filter pushdown
>>> - AWS module is added to support better integration with AWS, with AWS
>>> Glue catalog <https://aws.amazon.com/glue> support and dedicated S3
>>> FileIO implementation
>>> - Nessie module is added to support integration with project Nessie
>>> <https://projectnessie.org/>
>>>
>>> *Important bug fixes*
>>>
>>> - #1981 fixes date and timestamp transforms
>>> - #2091 fixes Parquet vectorized reads when column types are promoted
>>> - #1962 fixes Parquet vectorized position reader
>>> - #1991 fixes Avro schema conversions to preserve field docs
>>> - #1811 makes refreshing Spark cache optional
>>> - #1798 fixes read failure when encountering duplicate entries of
>>> data files
>>> - #1785 fixes invalidation of metadata tables in CachingCatalog
>>> - #1784 fixes resolving of SparkSession table's metadata tables
>>>
>>> *Other notable changes*
>>>
>>> - NaN counter is added to format v2 metrics
>>> - Shared catalog properties are added in core library to standardize
>>> catalog level configurations
>>> - Spark and Flink now supports dynamically loading customized
>>> `Catalog` and `FileIO` implementations
>>> - Spark now supports loading tables with file paths via HadoopTables
>>> - Spark 2 now supports loading tables from other catalogs, like
>>> Spark 3
>>> - Spark 3 now supports catalog names in DataFrameReader when using
>>> Iceberg as a format
>>> - Hive now supports INSERT INTO, case insensitive query, projection
>>> pushdown, create DDL with schema and auto type conversion
>>> - ORC now supports reading tinyint, smallint, char, varchar types
>>> - Hadoop catalog now supports role-based access of table listing
>>>
>>> Please download, verify, and test.
>>>
>>> Please vote in the next 72 hours.
>>>
>>> [ ] +1 Release this as Apache Iceberg 0.11.0
>>> [ ] +0
>>> [ ] -1 Do not release this because...
>>>
>>>
>>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
OpenInx, we can update documentation any time. It should not block the
release, just open a PR and we'll merge it and push docs after that.
On Mon, Jan 25, 2021 at 9:01 PM OpenInx <op...@gmail.com> wrote:
> Hi dev
>
> I'd like to include this patch in release 0.11.0 because it's the
> document of new flink features. I'm sorry that I did not update the
> flink's document in time when the feature code merged, but I think it's
> worth it to merge this document PR when we release iceberg 0.11.0, that
> helps a lot for users who want to use those new features, such as streaming
> reader, rewrite data files action, write distribution to cluster data.
> (I will update those documents in time so that we won't rollback the next
> release).
>
> Thanks.
>
>
> On Tue, Jan 26, 2021 at 10:17 AM Anton Okolnychyi
> <ao...@apple.com.invalid> wrote:
>
>> +1 (binding)
>>
>> I did local tests with Spark 3.0.1. I think we should also note the
>> support for DELETE FROM and MERGE INTO in Spark is experimental.
>>
>> Thanks,
>> Anton
>>
>> On 22 Jan 2021, at 15:26, Jack Ye <ye...@gmail.com> wrote:
>>
>> Hi everyone,
>>
>> I propose the following RC to be released as the official Apache Iceberg
>> 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>>
>> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>>
>> This corresponds to the tag: apache-iceberg-0.11.0-rc0
>> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
>> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
>>
>> The release tarball, signature, and checksums are here:
>> *
>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
>>
>> You can find the KEYS file here:
>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>
>> Convenience binary artifacts are staged in Nexus. The Maven repository
>> URL is:
>> *
>> https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>>
>> This release includes the following changes:
>>
>> *High-level features*
>>
>> - Core API now supports partition spec and sort order evolution
>> - Spark 3 now supports the following SQL extensions:
>> - MERGE INTO
>> - DELETE FROM
>> - ALTER TABLE ... ADD/DROP PARTITION
>> - ALTER TABLE ... WRITE ORDERED BY
>> - invoke stored procedures using CALL
>> - Flink now supports streaming reads, CDC writes (experimental), and
>> filter pushdown
>> - AWS module is added to support better integration with AWS, with AWS
>> Glue catalog <https://aws.amazon.com/glue> support and dedicated S3
>> FileIO implementation
>> - Nessie module is added to support integration with project Nessie
>> <https://projectnessie.org/>
>>
>> *Important bug fixes*
>>
>> - #1981 fixes date and timestamp transforms
>> - #2091 fixes Parquet vectorized reads when column types are promoted
>> - #1962 fixes Parquet vectorized position reader
>> - #1991 fixes Avro schema conversions to preserve field docs
>> - #1811 makes refreshing Spark cache optional
>> - #1798 fixes read failure when encountering duplicate entries of
>> data files
>> - #1785 fixes invalidation of metadata tables in CachingCatalog
>> - #1784 fixes resolving of SparkSession table's metadata tables
>>
>> *Other notable changes*
>>
>> - NaN counter is added to format v2 metrics
>> - Shared catalog properties are added in core library to standardize
>> catalog level configurations
>> - Spark and Flink now supports dynamically loading customized
>> `Catalog` and `FileIO` implementations
>> - Spark now supports loading tables with file paths via HadoopTables
>> - Spark 2 now supports loading tables from other catalogs, like Spark
>> 3
>> - Spark 3 now supports catalog names in DataFrameReader when using
>> Iceberg as a format
>> - Hive now supports INSERT INTO, case insensitive query, projection
>> pushdown, create DDL with schema and auto type conversion
>> - ORC now supports reading tinyint, smallint, char, varchar types
>> - Hadoop catalog now supports role-based access of table listing
>>
>> Please download, verify, and test.
>>
>> Please vote in the next 72 hours.
>>
>> [ ] +1 Release this as Apache Iceberg 0.11.0
>> [ ] +0
>> [ ] -1 Do not release this because...
>>
>>
>>
--
Ryan Blue
Software Engineer
Netflix
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by OpenInx <op...@gmail.com>.
Hi dev
I'd like to include this patch in release 0.11.0 because it's the document
of new flink features. I'm sorry that I did not update the flink's
document in time when the feature code merged, but I think it's worth it to
merge this document PR when we release iceberg 0.11.0, that helps a lot for
users who want to use those new features, such as streaming reader,
rewrite data files action, write distribution to cluster data. (I will
update those documents in time so that we won't rollback the next release).
Thanks.
On Tue, Jan 26, 2021 at 10:17 AM Anton Okolnychyi
<ao...@apple.com.invalid> wrote:
> +1 (binding)
>
> I did local tests with Spark 3.0.1. I think we should also note the
> support for DELETE FROM and MERGE INTO in Spark is experimental.
>
> Thanks,
> Anton
>
> On 22 Jan 2021, at 15:26, Jack Ye <ye...@gmail.com> wrote:
>
> Hi everyone,
>
> I propose the following RC to be released as the official Apache Iceberg
> 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>
> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>
> This corresponds to the tag: apache-iceberg-0.11.0-rc0
> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
>
> The release tarball, signature, and checksums are here:
> * https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>
> Convenience binary artifacts are staged in Nexus. The Maven repository URL
> is:
> * https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>
> This release includes the following changes:
>
> *High-level features*
>
> - Core API now supports partition spec and sort order evolution
> - Spark 3 now supports the following SQL extensions:
> - MERGE INTO
> - DELETE FROM
> - ALTER TABLE ... ADD/DROP PARTITION
> - ALTER TABLE ... WRITE ORDERED BY
> - invoke stored procedures using CALL
> - Flink now supports streaming reads, CDC writes (experimental), and
> filter pushdown
> - AWS module is added to support better integration with AWS, with AWS
> Glue catalog <https://aws.amazon.com/glue> support and dedicated S3
> FileIO implementation
> - Nessie module is added to support integration with project Nessie
> <https://projectnessie.org/>
>
> *Important bug fixes*
>
> - #1981 fixes date and timestamp transforms
> - #2091 fixes Parquet vectorized reads when column types are promoted
> - #1962 fixes Parquet vectorized position reader
> - #1991 fixes Avro schema conversions to preserve field docs
> - #1811 makes refreshing Spark cache optional
> - #1798 fixes read failure when encountering duplicate entries of data
> files
> - #1785 fixes invalidation of metadata tables in CachingCatalog
> - #1784 fixes resolving of SparkSession table's metadata tables
>
> *Other notable changes*
>
> - NaN counter is added to format v2 metrics
> - Shared catalog properties are added in core library to standardize
> catalog level configurations
> - Spark and Flink now supports dynamically loading customized
> `Catalog` and `FileIO` implementations
> - Spark now supports loading tables with file paths via HadoopTables
> - Spark 2 now supports loading tables from other catalogs, like Spark 3
> - Spark 3 now supports catalog names in DataFrameReader when using
> Iceberg as a format
> - Hive now supports INSERT INTO, case insensitive query, projection
> pushdown, create DDL with schema and auto type conversion
> - ORC now supports reading tinyint, smallint, char, varchar types
> - Hadoop catalog now supports role-based access of table listing
>
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as Apache Iceberg 0.11.0
> [ ] +0
> [ ] -1 Do not release this because...
>
>
>
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by Anton Okolnychyi <ao...@apple.com.INVALID>.
+1 (binding)
I did local tests with Spark 3.0.1. I think we should also note the support for DELETE FROM and MERGE INTO in Spark is experimental.
Thanks,
Anton
> On 22 Jan 2021, at 15:26, Jack Ye <ye...@gmail.com> wrote:
>
> Hi everyone,
>
> I propose the following RC to be released as the official Apache Iceberg 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>
> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>
> This corresponds to the tag: apache-iceberg-0.11.0-rc0
> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0 <https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0>
> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0 <https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0>
>
> The release tarball, signature, and checksums are here:
> * https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0 <https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0>
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS <https://dist.apache.org/repos/dist/dev/iceberg/KEYS>
>
> Convenience binary artifacts are staged in Nexus. The Maven repository URL is:
> * https://repository.apache.org/content/repositories/orgapacheiceberg-1015 <https://repository.apache.org/content/repositories/orgapacheiceberg-1015>
>
> This release includes the following changes:
>
> High-level features
> Core API now supports partition spec and sort order evolution
> Spark 3 now supports the following SQL extensions:
> MERGE INTO
> DELETE FROM
> ALTER TABLE ... ADD/DROP PARTITION
> ALTER TABLE ... WRITE ORDERED BY
> invoke stored procedures using CALL
> Flink now supports streaming reads, CDC writes (experimental), and filter pushdown
> AWS module is added to support better integration with AWS, with AWS Glue catalog <https://aws.amazon.com/glue> support and dedicated S3 FileIO implementation
> Nessie module is added to support integration with project Nessie <https://projectnessie.org/>
> Important bug fixes
> #1981 fixes date and timestamp transforms
> #2091 fixes Parquet vectorized reads when column types are promoted
> #1962 fixes Parquet vectorized position reader
> #1991 fixes Avro schema conversions to preserve field docs
> #1811 makes refreshing Spark cache optional
> #1798 fixes read failure when encountering duplicate entries of data files
> #1785 fixes invalidation of metadata tables in CachingCatalog
> #1784 fixes resolving of SparkSession table's metadata tables
> Other notable changes
> NaN counter is added to format v2 metrics
> Shared catalog properties are added in core library to standardize catalog level configurations
> Spark and Flink now supports dynamically loading customized `Catalog` and `FileIO` implementations
> Spark now supports loading tables with file paths via HadoopTables
> Spark 2 now supports loading tables from other catalogs, like Spark 3
> Spark 3 now supports catalog names in DataFrameReader when using Iceberg as a format
> Hive now supports INSERT INTO, case insensitive query, projection pushdown, create DDL with schema and auto type conversion
> ORC now supports reading tinyint, smallint, char, varchar types
> Hadoop catalog now supports role-based access of table listing
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as Apache Iceberg 0.11.0
> [ ] +0
> [ ] -1 Do not release this because...
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by Jack Ye <ye...@gmail.com>.
That issue is closed, it looks like just a user error.
On Sun, Jan 24, 2021 at 12:07 PM Ryan Murray <ry...@dremio.com> wrote:
> Based on #2144 <https://github.com/apache/iceberg/issues/2144> I would
> change my vote to -1. I will raise a fix asap.
>
>
> On Sun, Jan 24, 2021 at 3:12 PM Ryan Murray <ry...@dremio.com> wrote:
>
>>
>> +1 (non-binding)
>>
>> I verified the build and ran the tests. Also verified both flink and
>> spark custom catalogs are working.
>>
>> One side note: I had to run the tests a few times to get the build to
>> pass. Flaky Hive tests.
>>
>> Best,
>>
>> Ryan Murray
>>
>> On Sat, Jan 23, 2021 at 12:26 AM Jack Ye <ye...@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> I propose the following RC to be released as the official Apache Iceberg
>>> 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>>>
>>> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>>>
>>> This corresponds to the tag: apache-iceberg-0.11.0-rc0
>>> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
>>> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
>>>
>>> The release tarball, signature, and checksums are here:
>>> *
>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
>>>
>>> You can find the KEYS file here:
>>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>>
>>> Convenience binary artifacts are staged in Nexus. The Maven repository
>>> URL is:
>>> *
>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>>>
>>> This release includes the following changes:
>>>
>>> *High-level features*
>>>
>>> - Core API now supports partition spec and sort order evolution
>>> - Spark 3 now supports the following SQL extensions:
>>> - MERGE INTO
>>> - DELETE FROM
>>> - ALTER TABLE ... ADD/DROP PARTITION
>>> - ALTER TABLE ... WRITE ORDERED BY
>>> - invoke stored procedures using CALL
>>> - Flink now supports streaming reads, CDC writes (experimental), and
>>> filter pushdown
>>> - AWS module is added to support better integration with AWS, with AWS
>>> Glue catalog <https://aws.amazon.com/glue> support and dedicated S3
>>> FileIO implementation
>>> - Nessie module is added to support integration with project Nessie
>>> <https://projectnessie.org>
>>>
>>> *Important bug fixes*
>>>
>>> - #1981 fixes date and timestamp transforms
>>> - #2091 fixes Parquet vectorized reads when column types are promoted
>>> - #1962 fixes Parquet vectorized position reader
>>> - #1991 fixes Avro schema conversions to preserve field docs
>>> - #1811 makes refreshing Spark cache optional
>>> - #1798 fixes read failure when encountering duplicate entries of
>>> data files
>>> - #1785 fixes invalidation of metadata tables in CachingCatalog
>>> - #1784 fixes resolving of SparkSession table's metadata tables
>>>
>>> *Other notable changes*
>>>
>>> - NaN counter is added to format v2 metrics
>>> - Shared catalog properties are added in core library to standardize
>>> catalog level configurations
>>> - Spark and Flink now supports dynamically loading customized
>>> `Catalog` and `FileIO` implementations
>>> - Spark now supports loading tables with file paths via HadoopTables
>>> - Spark 2 now supports loading tables from other catalogs, like
>>> Spark 3
>>> - Spark 3 now supports catalog names in DataFrameReader when using
>>> Iceberg as a format
>>> - Hive now supports INSERT INTO, case insensitive query, projection
>>> pushdown, create DDL with schema and auto type conversion
>>> - ORC now supports reading tinyint, smallint, char, varchar types
>>> - Hadoop catalog now supports role-based access of table listing
>>>
>>> Please download, verify, and test.
>>>
>>> Please vote in the next 72 hours.
>>>
>>> [ ] +1 Release this as Apache Iceberg 0.11.0
>>> [ ] +0
>>> [ ] -1 Do not release this because...
>>>
>>
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by Ryan Murray <ry...@dremio.com>.
I agree. I saw similar errors while testing this afternoon. Will do more
testing tomorrow but I think it's ok...
On Sun, 24 Jan 2021, 22:01 Ryan Blue, <rb...@netflix.com.invalid> wrote:
> Looks like the problem in #2144 is that the error message isn't quite
> right. The problem was that `hadoop` was used for the Catalog
> implementation rather than the type, which is a short-cut to not need to
> specify the full class name. The error message should be better and note
> that the class was actually not found, instead of the no-arg constructor.
> But I don't think that's a problem that needs to be fixed for the release.
>
> On Sun, Jan 24, 2021 at 12:07 PM Ryan Murray <ry...@dremio.com> wrote:
>
>> Based on #2144 <https://github.com/apache/iceberg/issues/2144> I would
>> change my vote to -1. I will raise a fix asap.
>>
>>
>> On Sun, Jan 24, 2021 at 3:12 PM Ryan Murray <ry...@dremio.com> wrote:
>>
>>>
>>> +1 (non-binding)
>>>
>>> I verified the build and ran the tests. Also verified both flink and
>>> spark custom catalogs are working.
>>>
>>> One side note: I had to run the tests a few times to get the build to
>>> pass. Flaky Hive tests.
>>>
>>> Best,
>>>
>>> Ryan Murray
>>>
>>> On Sat, Jan 23, 2021 at 12:26 AM Jack Ye <ye...@gmail.com> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I propose the following RC to be released as the official Apache
>>>> Iceberg 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>>>>
>>>> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>>>>
>>>> This corresponds to the tag: apache-iceberg-0.11.0-rc0
>>>> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
>>>> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
>>>>
>>>> The release tarball, signature, and checksums are here:
>>>> *
>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
>>>>
>>>> You can find the KEYS file here:
>>>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>>>
>>>> Convenience binary artifacts are staged in Nexus. The Maven repository
>>>> URL is:
>>>> *
>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>>>>
>>>> This release includes the following changes:
>>>>
>>>> *High-level features*
>>>>
>>>> - Core API now supports partition spec and sort order evolution
>>>> - Spark 3 now supports the following SQL extensions:
>>>> - MERGE INTO
>>>> - DELETE FROM
>>>> - ALTER TABLE ... ADD/DROP PARTITION
>>>> - ALTER TABLE ... WRITE ORDERED BY
>>>> - invoke stored procedures using CALL
>>>> - Flink now supports streaming reads, CDC writes (experimental),
>>>> and filter pushdown
>>>> - AWS module is added to support better integration with AWS, with AWS
>>>> Glue catalog <https://aws.amazon.com/glue> support and dedicated S3
>>>> FileIO implementation
>>>> - Nessie module is added to support integration with project Nessie
>>>> <https://projectnessie.org>
>>>>
>>>> *Important bug fixes*
>>>>
>>>> - #1981 fixes date and timestamp transforms
>>>> - #2091 fixes Parquet vectorized reads when column types are
>>>> promoted
>>>> - #1962 fixes Parquet vectorized position reader
>>>> - #1991 fixes Avro schema conversions to preserve field docs
>>>> - #1811 makes refreshing Spark cache optional
>>>> - #1798 fixes read failure when encountering duplicate entries of
>>>> data files
>>>> - #1785 fixes invalidation of metadata tables in CachingCatalog
>>>> - #1784 fixes resolving of SparkSession table's metadata tables
>>>>
>>>> *Other notable changes*
>>>>
>>>> - NaN counter is added to format v2 metrics
>>>> - Shared catalog properties are added in core library to
>>>> standardize catalog level configurations
>>>> - Spark and Flink now supports dynamically loading customized
>>>> `Catalog` and `FileIO` implementations
>>>> - Spark now supports loading tables with file paths via HadoopTables
>>>> - Spark 2 now supports loading tables from other catalogs, like
>>>> Spark 3
>>>> - Spark 3 now supports catalog names in DataFrameReader when using
>>>> Iceberg as a format
>>>> - Hive now supports INSERT INTO, case insensitive query, projection
>>>> pushdown, create DDL with schema and auto type conversion
>>>> - ORC now supports reading tinyint, smallint, char, varchar types
>>>> - Hadoop catalog now supports role-based access of table listing
>>>>
>>>> Please download, verify, and test.
>>>>
>>>> Please vote in the next 72 hours.
>>>>
>>>> [ ] +1 Release this as Apache Iceberg 0.11.0
>>>> [ ] +0
>>>> [ ] -1 Do not release this because...
>>>>
>>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
Looks like the problem in #2144 is that the error message isn't quite
right. The problem was that `hadoop` was used for the Catalog
implementation rather than the type, which is a short-cut to not need to
specify the full class name. The error message should be better and note
that the class was actually not found, instead of the no-arg constructor.
But I don't think that's a problem that needs to be fixed for the release.
On Sun, Jan 24, 2021 at 12:07 PM Ryan Murray <ry...@dremio.com> wrote:
> Based on #2144 <https://github.com/apache/iceberg/issues/2144> I would
> change my vote to -1. I will raise a fix asap.
>
>
> On Sun, Jan 24, 2021 at 3:12 PM Ryan Murray <ry...@dremio.com> wrote:
>
>>
>> +1 (non-binding)
>>
>> I verified the build and ran the tests. Also verified both flink and
>> spark custom catalogs are working.
>>
>> One side note: I had to run the tests a few times to get the build to
>> pass. Flaky Hive tests.
>>
>> Best,
>>
>> Ryan Murray
>>
>> On Sat, Jan 23, 2021 at 12:26 AM Jack Ye <ye...@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> I propose the following RC to be released as the official Apache Iceberg
>>> 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>>>
>>> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>>>
>>> This corresponds to the tag: apache-iceberg-0.11.0-rc0
>>> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
>>> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
>>>
>>> The release tarball, signature, and checksums are here:
>>> *
>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
>>>
>>> You can find the KEYS file here:
>>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>>
>>> Convenience binary artifacts are staged in Nexus. The Maven repository
>>> URL is:
>>> *
>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>>>
>>> This release includes the following changes:
>>>
>>> *High-level features*
>>>
>>> - Core API now supports partition spec and sort order evolution
>>> - Spark 3 now supports the following SQL extensions:
>>> - MERGE INTO
>>> - DELETE FROM
>>> - ALTER TABLE ... ADD/DROP PARTITION
>>> - ALTER TABLE ... WRITE ORDERED BY
>>> - invoke stored procedures using CALL
>>> - Flink now supports streaming reads, CDC writes (experimental), and
>>> filter pushdown
>>> - AWS module is added to support better integration with AWS, with AWS
>>> Glue catalog <https://aws.amazon.com/glue> support and dedicated S3
>>> FileIO implementation
>>> - Nessie module is added to support integration with project Nessie
>>> <https://projectnessie.org>
>>>
>>> *Important bug fixes*
>>>
>>> - #1981 fixes date and timestamp transforms
>>> - #2091 fixes Parquet vectorized reads when column types are promoted
>>> - #1962 fixes Parquet vectorized position reader
>>> - #1991 fixes Avro schema conversions to preserve field docs
>>> - #1811 makes refreshing Spark cache optional
>>> - #1798 fixes read failure when encountering duplicate entries of
>>> data files
>>> - #1785 fixes invalidation of metadata tables in CachingCatalog
>>> - #1784 fixes resolving of SparkSession table's metadata tables
>>>
>>> *Other notable changes*
>>>
>>> - NaN counter is added to format v2 metrics
>>> - Shared catalog properties are added in core library to standardize
>>> catalog level configurations
>>> - Spark and Flink now supports dynamically loading customized
>>> `Catalog` and `FileIO` implementations
>>> - Spark now supports loading tables with file paths via HadoopTables
>>> - Spark 2 now supports loading tables from other catalogs, like
>>> Spark 3
>>> - Spark 3 now supports catalog names in DataFrameReader when using
>>> Iceberg as a format
>>> - Hive now supports INSERT INTO, case insensitive query, projection
>>> pushdown, create DDL with schema and auto type conversion
>>> - ORC now supports reading tinyint, smallint, char, varchar types
>>> - Hadoop catalog now supports role-based access of table listing
>>>
>>> Please download, verify, and test.
>>>
>>> Please vote in the next 72 hours.
>>>
>>> [ ] +1 Release this as Apache Iceberg 0.11.0
>>> [ ] +0
>>> [ ] -1 Do not release this because...
>>>
>>
--
Ryan Blue
Software Engineer
Netflix
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by Ryan Murray <ry...@dremio.com>.
Based on #2144 <https://github.com/apache/iceberg/issues/2144> I would
change my vote to -1. I will raise a fix asap.
On Sun, Jan 24, 2021 at 3:12 PM Ryan Murray <ry...@dremio.com> wrote:
>
> +1 (non-binding)
>
> I verified the build and ran the tests. Also verified both flink and spark
> custom catalogs are working.
>
> One side note: I had to run the tests a few times to get the build to
> pass. Flaky Hive tests.
>
> Best,
>
> Ryan Murray
>
> On Sat, Jan 23, 2021 at 12:26 AM Jack Ye <ye...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> I propose the following RC to be released as the official Apache Iceberg
>> 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>>
>> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>>
>> This corresponds to the tag: apache-iceberg-0.11.0-rc0
>> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
>> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
>>
>> The release tarball, signature, and checksums are here:
>> *
>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
>>
>> You can find the KEYS file here:
>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>>
>> Convenience binary artifacts are staged in Nexus. The Maven repository
>> URL is:
>> *
>> https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>>
>> This release includes the following changes:
>>
>> *High-level features*
>>
>> - Core API now supports partition spec and sort order evolution
>> - Spark 3 now supports the following SQL extensions:
>> - MERGE INTO
>> - DELETE FROM
>> - ALTER TABLE ... ADD/DROP PARTITION
>> - ALTER TABLE ... WRITE ORDERED BY
>> - invoke stored procedures using CALL
>> - Flink now supports streaming reads, CDC writes (experimental), and
>> filter pushdown
>> - AWS module is added to support better integration with AWS, with AWS
>> Glue catalog <https://aws.amazon.com/glue> support and dedicated S3
>> FileIO implementation
>> - Nessie module is added to support integration with project Nessie
>> <https://projectnessie.org>
>>
>> *Important bug fixes*
>>
>> - #1981 fixes date and timestamp transforms
>> - #2091 fixes Parquet vectorized reads when column types are promoted
>> - #1962 fixes Parquet vectorized position reader
>> - #1991 fixes Avro schema conversions to preserve field docs
>> - #1811 makes refreshing Spark cache optional
>> - #1798 fixes read failure when encountering duplicate entries of
>> data files
>> - #1785 fixes invalidation of metadata tables in CachingCatalog
>> - #1784 fixes resolving of SparkSession table's metadata tables
>>
>> *Other notable changes*
>>
>> - NaN counter is added to format v2 metrics
>> - Shared catalog properties are added in core library to standardize
>> catalog level configurations
>> - Spark and Flink now supports dynamically loading customized
>> `Catalog` and `FileIO` implementations
>> - Spark now supports loading tables with file paths via HadoopTables
>> - Spark 2 now supports loading tables from other catalogs, like Spark
>> 3
>> - Spark 3 now supports catalog names in DataFrameReader when using
>> Iceberg as a format
>> - Hive now supports INSERT INTO, case insensitive query, projection
>> pushdown, create DDL with schema and auto type conversion
>> - ORC now supports reading tinyint, smallint, char, varchar types
>> - Hadoop catalog now supports role-based access of table listing
>>
>> Please download, verify, and test.
>>
>> Please vote in the next 72 hours.
>>
>> [ ] +1 Release this as Apache Iceberg 0.11.0
>> [ ] +0
>> [ ] -1 Do not release this because...
>>
>
Re: [VOTE] Release Apache Iceberg 0.11.0 RC0
Posted by Ryan Murray <ry...@dremio.com>.
+1 (non-binding)
I verified the build and ran the tests. Also verified both flink and spark
custom catalogs are working.
One side note: I had to run the tests a few times to get the build to pass.
Flaky Hive tests.
Best,
Ryan Murray
On Sat, Jan 23, 2021 at 12:26 AM Jack Ye <ye...@gmail.com> wrote:
> Hi everyone,
>
> I propose the following RC to be released as the official Apache Iceberg
> 0.11.0 release. The RC is also reviewed and signed by Ryan Blue.
>
> The commit id is ad78cc6cf259b7a0c66ab5de6675cc005febd939
>
> This corresponds to the tag: apache-iceberg-0.11.0-rc0
> * https://github.com/apache/iceberg/commits/apache-iceberg-0.11.0-rc0
> * https://github.com/apache/iceberg/tree/apache-iceberg-0.11.0-rc0
>
> The release tarball, signature, and checksums are here:
> * https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS
>
> Convenience binary artifacts are staged in Nexus. The Maven repository URL
> is:
> * https://repository.apache.org/content/repositories/orgapacheiceberg-1015
>
> This release includes the following changes:
>
> *High-level features*
>
> - Core API now supports partition spec and sort order evolution
> - Spark 3 now supports the following SQL extensions:
> - MERGE INTO
> - DELETE FROM
> - ALTER TABLE ... ADD/DROP PARTITION
> - ALTER TABLE ... WRITE ORDERED BY
> - invoke stored procedures using CALL
> - Flink now supports streaming reads, CDC writes (experimental), and
> filter pushdown
> - AWS module is added to support better integration with AWS, with AWS
> Glue catalog <https://aws.amazon.com/glue> support and dedicated S3
> FileIO implementation
> - Nessie module is added to support integration with project Nessie
> <https://projectnessie.org>
>
> *Important bug fixes*
>
> - #1981 fixes date and timestamp transforms
> - #2091 fixes Parquet vectorized reads when column types are promoted
> - #1962 fixes Parquet vectorized position reader
> - #1991 fixes Avro schema conversions to preserve field docs
> - #1811 makes refreshing Spark cache optional
> - #1798 fixes read failure when encountering duplicate entries of data
> files
> - #1785 fixes invalidation of metadata tables in CachingCatalog
> - #1784 fixes resolving of SparkSession table's metadata tables
>
> *Other notable changes*
>
> - NaN counter is added to format v2 metrics
> - Shared catalog properties are added in core library to standardize
> catalog level configurations
> - Spark and Flink now supports dynamically loading customized
> `Catalog` and `FileIO` implementations
> - Spark now supports loading tables with file paths via HadoopTables
> - Spark 2 now supports loading tables from other catalogs, like Spark 3
> - Spark 3 now supports catalog names in DataFrameReader when using
> Iceberg as a format
> - Hive now supports INSERT INTO, case insensitive query, projection
> pushdown, create DDL with schema and auto type conversion
> - ORC now supports reading tinyint, smallint, char, varchar types
> - Hadoop catalog now supports role-based access of table listing
>
> Please download, verify, and test.
>
> Please vote in the next 72 hours.
>
> [ ] +1 Release this as Apache Iceberg 0.11.0
> [ ] +0
> [ ] -1 Do not release this because...
>