You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/01/26 21:38:30 UTC
[GitHub] [iceberg] jackye1995 opened a new pull request #2157: Doc: update 0.11 release notes
jackye1995 opened a new pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564860251
##########
File path: site/docs/releases.md
##########
@@ -51,44 +51,60 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`:
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-core</artifactId>
- <version>0.10.0</version>
+ <version>0.11.0</version>
</dependency>
...
</dependencies>
```
-## 0.10.0 release notes
+## 0.11.0 release notes
High-level features:
-* **Format v2 support** for building row-level operations (`MERGE INTO`) in processing engines
- * Note: format v2 is not yet finalized and does not have a forward-compatibility guarantee
-* **Flink integration** for writing to Iceberg tables and reading from Iceberg tables (reading supports batch mode only)
-* **Hive integration** for reading from Iceberg tables, with filter pushdown (experimental; configuration may change)
+* **Core API** now supports partition spec and sort order evolution
+* **Spark 3** now supports the following SQL extensions:
+ * MERGE INTO (experimental)
+ * DELETE FROM (experimental)
+ * ALTER TABLE ... ADD/DROP PARTITION
+ * ALTER TABLE ... WRITE ORDERED BY
+ * invoke stored procedures using CALL
+* **Flink** now supports streaming reads, CDC writes (experimental), and filter pushdown
+* **AWS module** is added to support better integration with AWS, with AWS Glue catalog support and dedicated S3 FileIO implementation
+* **Nessie module** is added to support integration with project Nessie
Important bug fixes:
-* [\#1706](https://github.com/apache/iceberg/pull/1706) fixes non-vectorized ORC reads in Spark that incorrectly skipped rows
-* [\#1536](https://github.com/apache/iceberg/pull/1536) fixes ORC conversion of `notIn` and `notEqual` to match null values
-* [\#1722](https://github.com/apache/iceberg/pull/1722) fixes `Expressions.notNull` returning an `isNull` predicate; API only, method was not used by processing engines
-* [\#1736](https://github.com/apache/iceberg/pull/1736) fixes `IllegalArgumentException` in vectorized Spark reads with negative decimal values
-* [\#1666](https://github.com/apache/iceberg/pull/1666) fixes file lengths returned by the ORC writer, using compressed size rather than uncompressed size
-* [\#1674](https://github.com/apache/iceberg/pull/1674) removes catalog expiration in HiveCatalogs
-* [\#1545](https://github.com/apache/iceberg/pull/1545) automatically refreshes tables in Spark when not caching table instances
+* [\#2091](https://github.com/apache/iceberg/pull/2091) fixes Parquet vectorized reads when column types are promoted
+* [\#1991](https://github.com/apache/iceberg/pull/1991) fixes Avro schema conversions to preserve field docs
+* [\#1981](https://github.com/apache/iceberg/pull/1981) fixes date and timestamp transforms
+* [\#1962](https://github.com/apache/iceberg/pull/1962) fixes Parquet vectorized position reader
+* [\#1811](https://github.com/apache/iceberg/pull/1811) makes refreshing Spark cache optional
Review comment:
This also fixes a new feature, so it isn't a relevant bug fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564858846
##########
File path: site/docs/releases.md
##########
@@ -51,44 +51,60 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`:
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-core</artifactId>
- <version>0.10.0</version>
+ <version>0.11.0</version>
</dependency>
...
</dependencies>
```
-## 0.10.0 release notes
+## 0.11.0 release notes
High-level features:
-* **Format v2 support** for building row-level operations (`MERGE INTO`) in processing engines
- * Note: format v2 is not yet finalized and does not have a forward-compatibility guarantee
-* **Flink integration** for writing to Iceberg tables and reading from Iceberg tables (reading supports batch mode only)
-* **Hive integration** for reading from Iceberg tables, with filter pushdown (experimental; configuration may change)
+* **Core API** now supports partition spec and sort order evolution
+* **Spark 3** now supports the following SQL extensions:
+ * MERGE INTO (experimental)
+ * DELETE FROM (experimental)
+ * ALTER TABLE ... ADD/DROP PARTITION
+ * ALTER TABLE ... WRITE ORDERED BY
+ * invoke stored procedures using CALL
+* **Flink** now supports streaming reads, CDC writes (experimental), and filter pushdown
+* **AWS module** is added to support better integration with AWS, with AWS Glue catalog support and dedicated S3 FileIO implementation
+* **Nessie module** is added to support integration with project Nessie
Important bug fixes:
-* [\#1706](https://github.com/apache/iceberg/pull/1706) fixes non-vectorized ORC reads in Spark that incorrectly skipped rows
-* [\#1536](https://github.com/apache/iceberg/pull/1536) fixes ORC conversion of `notIn` and `notEqual` to match null values
-* [\#1722](https://github.com/apache/iceberg/pull/1722) fixes `Expressions.notNull` returning an `isNull` predicate; API only, method was not used by processing engines
-* [\#1736](https://github.com/apache/iceberg/pull/1736) fixes `IllegalArgumentException` in vectorized Spark reads with negative decimal values
-* [\#1666](https://github.com/apache/iceberg/pull/1666) fixes file lengths returned by the ORC writer, using compressed size rather than uncompressed size
-* [\#1674](https://github.com/apache/iceberg/pull/1674) removes catalog expiration in HiveCatalogs
-* [\#1545](https://github.com/apache/iceberg/pull/1545) automatically refreshes tables in Spark when not caching table instances
+* [\#2091](https://github.com/apache/iceberg/pull/2091) fixes Parquet vectorized reads when column types are promoted
+* [\#1991](https://github.com/apache/iceberg/pull/1991) fixes Avro schema conversions to preserve field docs
+* [\#1981](https://github.com/apache/iceberg/pull/1981) fixes date and timestamp transforms
+* [\#1962](https://github.com/apache/iceberg/pull/1962) fixes Parquet vectorized position reader
+* [\#1811](https://github.com/apache/iceberg/pull/1811) makes refreshing Spark cache optional
+* [\#1798](https://github.com/apache/iceberg/pull/1798) fixes read failure when encountering duplicate entries of data files
+* [\#1785](https://github.com/apache/iceberg/pull/1785) fixes invalidation of metadata tables in CachingCatalog
+* [\#1784](https://github.com/apache/iceberg/pull/1784) fixes resolving of SparkSession table's metadata tables
Other notable changes:
-* The `iceberg-hive` module has been renamed to `iceberg-hive-metastore` to avoid confusion
-* Spark 3 is based on 3.0.1 that includes the fix for [SPARK-32168](https://issues.apache.org/jira/browse/SPARK-32168)
-* Hadoop tables will recover from version hint corruption
-* Tables can be configured with a required sort order
-* Data file locations can be customized with a dynamically loaded `LocationProvider`
-* ORC file imports can apply a name mapping for stats
+* NaN counter is added to format v2 metrics
+* Shared catalog properties are added in core library to standardize catalog level configurations
+* Spark and Flink now supports dynamically loading customized `Catalog` and `FileIO` implementations
+* Spark now supports loading tables with file paths via HadoopTables
+* Spark 2 now supports loading tables from other catalogs, like Spark 3
+* Spark 3 now supports catalog names in DataFrameReader when using Iceberg as a format
+* Hive now supports INSERT INTO, case insensitive query, projection pushdown, create DDL with schema and auto type conversion
+* ORC now supports reading tinyint, smallint, char, varchar types
+* Hadoop catalog now supports role-based access of table listing
+## Past releases
-A more exhaustive list of changes is available under the [0.10.0 release milestone](https://github.com/apache/iceberg/milestone/10?closed=1).
+### 0.10.0
Review comment:
Should we keep the release notes for 0.10.0? Seems like a good thing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564860092
##########
File path: site/docs/releases.md
##########
@@ -51,44 +51,60 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`:
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-core</artifactId>
- <version>0.10.0</version>
+ <version>0.11.0</version>
</dependency>
...
</dependencies>
```
-## 0.10.0 release notes
+## 0.11.0 release notes
High-level features:
-* **Format v2 support** for building row-level operations (`MERGE INTO`) in processing engines
- * Note: format v2 is not yet finalized and does not have a forward-compatibility guarantee
-* **Flink integration** for writing to Iceberg tables and reading from Iceberg tables (reading supports batch mode only)
-* **Hive integration** for reading from Iceberg tables, with filter pushdown (experimental; configuration may change)
+* **Core API** now supports partition spec and sort order evolution
+* **Spark 3** now supports the following SQL extensions:
+ * MERGE INTO (experimental)
+ * DELETE FROM (experimental)
+ * ALTER TABLE ... ADD/DROP PARTITION
+ * ALTER TABLE ... WRITE ORDERED BY
+ * invoke stored procedures using CALL
+* **Flink** now supports streaming reads, CDC writes (experimental), and filter pushdown
+* **AWS module** is added to support better integration with AWS, with AWS Glue catalog support and dedicated S3 FileIO implementation
+* **Nessie module** is added to support integration with project Nessie
Important bug fixes:
-* [\#1706](https://github.com/apache/iceberg/pull/1706) fixes non-vectorized ORC reads in Spark that incorrectly skipped rows
-* [\#1536](https://github.com/apache/iceberg/pull/1536) fixes ORC conversion of `notIn` and `notEqual` to match null values
-* [\#1722](https://github.com/apache/iceberg/pull/1722) fixes `Expressions.notNull` returning an `isNull` predicate; API only, method was not used by processing engines
-* [\#1736](https://github.com/apache/iceberg/pull/1736) fixes `IllegalArgumentException` in vectorized Spark reads with negative decimal values
-* [\#1666](https://github.com/apache/iceberg/pull/1666) fixes file lengths returned by the ORC writer, using compressed size rather than uncompressed size
-* [\#1674](https://github.com/apache/iceberg/pull/1674) removes catalog expiration in HiveCatalogs
-* [\#1545](https://github.com/apache/iceberg/pull/1545) automatically refreshes tables in Spark when not caching table instances
+* [\#2091](https://github.com/apache/iceberg/pull/2091) fixes Parquet vectorized reads when column types are promoted
+* [\#1991](https://github.com/apache/iceberg/pull/1991) fixes Avro schema conversions to preserve field docs
+* [\#1981](https://github.com/apache/iceberg/pull/1981) fixes date and timestamp transforms
+* [\#1962](https://github.com/apache/iceberg/pull/1962) fixes Parquet vectorized position reader
Review comment:
We can remove this because the vectorized position reader is new.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564861365
##########
File path: site/docs/releases.md
##########
@@ -51,44 +51,60 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`:
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-core</artifactId>
- <version>0.10.0</version>
+ <version>0.11.0</version>
</dependency>
...
</dependencies>
```
-## 0.10.0 release notes
+## 0.11.0 release notes
High-level features:
-* **Format v2 support** for building row-level operations (`MERGE INTO`) in processing engines
- * Note: format v2 is not yet finalized and does not have a forward-compatibility guarantee
-* **Flink integration** for writing to Iceberg tables and reading from Iceberg tables (reading supports batch mode only)
-* **Hive integration** for reading from Iceberg tables, with filter pushdown (experimental; configuration may change)
+* **Core API** now supports partition spec and sort order evolution
+* **Spark 3** now supports the following SQL extensions:
+ * MERGE INTO (experimental)
+ * DELETE FROM (experimental)
+ * ALTER TABLE ... ADD/DROP PARTITION
+ * ALTER TABLE ... WRITE ORDERED BY
+ * invoke stored procedures using CALL
+* **Flink** now supports streaming reads, CDC writes (experimental), and filter pushdown
+* **AWS module** is added to support better integration with AWS, with AWS Glue catalog support and dedicated S3 FileIO implementation
+* **Nessie module** is added to support integration with project Nessie
Important bug fixes:
-* [\#1706](https://github.com/apache/iceberg/pull/1706) fixes non-vectorized ORC reads in Spark that incorrectly skipped rows
-* [\#1536](https://github.com/apache/iceberg/pull/1536) fixes ORC conversion of `notIn` and `notEqual` to match null values
-* [\#1722](https://github.com/apache/iceberg/pull/1722) fixes `Expressions.notNull` returning an `isNull` predicate; API only, method was not used by processing engines
-* [\#1736](https://github.com/apache/iceberg/pull/1736) fixes `IllegalArgumentException` in vectorized Spark reads with negative decimal values
-* [\#1666](https://github.com/apache/iceberg/pull/1666) fixes file lengths returned by the ORC writer, using compressed size rather than uncompressed size
-* [\#1674](https://github.com/apache/iceberg/pull/1674) removes catalog expiration in HiveCatalogs
-* [\#1545](https://github.com/apache/iceberg/pull/1545) automatically refreshes tables in Spark when not caching table instances
+* [\#2091](https://github.com/apache/iceberg/pull/2091) fixes Parquet vectorized reads when column types are promoted
+* [\#1991](https://github.com/apache/iceberg/pull/1991) fixes Avro schema conversions to preserve field docs
+* [\#1981](https://github.com/apache/iceberg/pull/1981) fixes date and timestamp transforms
+* [\#1962](https://github.com/apache/iceberg/pull/1962) fixes Parquet vectorized position reader
+* [\#1811](https://github.com/apache/iceberg/pull/1811) makes refreshing Spark cache optional
+* [\#1798](https://github.com/apache/iceberg/pull/1798) fixes read failure when encountering duplicate entries of data files
+* [\#1785](https://github.com/apache/iceberg/pull/1785) fixes invalidation of metadata tables in CachingCatalog
+* [\#1784](https://github.com/apache/iceberg/pull/1784) fixes resolving of SparkSession table's metadata tables
Other notable changes:
-* The `iceberg-hive` module has been renamed to `iceberg-hive-metastore` to avoid confusion
-* Spark 3 is based on 3.0.1 that includes the fix for [SPARK-32168](https://issues.apache.org/jira/browse/SPARK-32168)
-* Hadoop tables will recover from version hint corruption
-* Tables can be configured with a required sort order
-* Data file locations can be customized with a dynamically loaded `LocationProvider`
-* ORC file imports can apply a name mapping for stats
+* NaN counter is added to format v2 metrics
+* Shared catalog properties are added in core library to standardize catalog level configurations
+* Spark and Flink now supports dynamically loading customized `Catalog` and `FileIO` implementations
Review comment:
Typo: should be "support" not "supports" because the subject, "Spark and Flink" is more than one thing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564851737
##########
File path: site/docs/javadoc/index.html
##########
@@ -1,9 +1,9 @@
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Iceberg Javadoc Redirect</title>
- <meta http-equiv="refresh" content="0;URL='/javadoc/0.10.0/'" />
+ <meta http-equiv="refresh" content="0;URL='/javadoc/0.11.0/'" />
</head>
<body>
- <p>Redirecting to Javadoc for the 0.10.0 release: <a href="/javadoc/0.10.0/">/javadoc/0.10.0</a>.</p>
+ <p>Redirecting to Javadoc for the 0.11.0 release: <a href="/javadoc/0.11.0/">/javadoc/0.11.0</a>.</p>
Review comment:
Not sure how javadoc generated, is there anything that has to be manually done to have the 0.11 javadoc folder?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564859803
##########
File path: site/docs/releases.md
##########
@@ -51,44 +51,60 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`:
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-core</artifactId>
- <version>0.10.0</version>
+ <version>0.11.0</version>
</dependency>
...
</dependencies>
```
-## 0.10.0 release notes
+## 0.11.0 release notes
High-level features:
-* **Format v2 support** for building row-level operations (`MERGE INTO`) in processing engines
- * Note: format v2 is not yet finalized and does not have a forward-compatibility guarantee
-* **Flink integration** for writing to Iceberg tables and reading from Iceberg tables (reading supports batch mode only)
-* **Hive integration** for reading from Iceberg tables, with filter pushdown (experimental; configuration may change)
+* **Core API** now supports partition spec and sort order evolution
+* **Spark 3** now supports the following SQL extensions:
+ * MERGE INTO (experimental)
+ * DELETE FROM (experimental)
+ * ALTER TABLE ... ADD/DROP PARTITION
+ * ALTER TABLE ... WRITE ORDERED BY
+ * invoke stored procedures using CALL
Review comment:
Nit: capitalize invoke.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564864002
##########
File path: site/docs/releases.md
##########
@@ -51,44 +51,60 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`:
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-core</artifactId>
- <version>0.10.0</version>
+ <version>0.11.0</version>
</dependency>
...
</dependencies>
```
-## 0.10.0 release notes
+## 0.11.0 release notes
High-level features:
-* **Format v2 support** for building row-level operations (`MERGE INTO`) in processing engines
- * Note: format v2 is not yet finalized and does not have a forward-compatibility guarantee
-* **Flink integration** for writing to Iceberg tables and reading from Iceberg tables (reading supports batch mode only)
-* **Hive integration** for reading from Iceberg tables, with filter pushdown (experimental; configuration may change)
+* **Core API** now supports partition spec and sort order evolution
+* **Spark 3** now supports the following SQL extensions:
+ * MERGE INTO (experimental)
+ * DELETE FROM (experimental)
+ * ALTER TABLE ... ADD/DROP PARTITION
+ * ALTER TABLE ... WRITE ORDERED BY
+ * invoke stored procedures using CALL
+* **Flink** now supports streaming reads, CDC writes (experimental), and filter pushdown
+* **AWS module** is added to support better integration with AWS, with AWS Glue catalog support and dedicated S3 FileIO implementation
+* **Nessie module** is added to support integration with project Nessie
Important bug fixes:
-* [\#1706](https://github.com/apache/iceberg/pull/1706) fixes non-vectorized ORC reads in Spark that incorrectly skipped rows
-* [\#1536](https://github.com/apache/iceberg/pull/1536) fixes ORC conversion of `notIn` and `notEqual` to match null values
-* [\#1722](https://github.com/apache/iceberg/pull/1722) fixes `Expressions.notNull` returning an `isNull` predicate; API only, method was not used by processing engines
-* [\#1736](https://github.com/apache/iceberg/pull/1736) fixes `IllegalArgumentException` in vectorized Spark reads with negative decimal values
-* [\#1666](https://github.com/apache/iceberg/pull/1666) fixes file lengths returned by the ORC writer, using compressed size rather than uncompressed size
-* [\#1674](https://github.com/apache/iceberg/pull/1674) removes catalog expiration in HiveCatalogs
-* [\#1545](https://github.com/apache/iceberg/pull/1545) automatically refreshes tables in Spark when not caching table instances
+* [\#2091](https://github.com/apache/iceberg/pull/2091) fixes Parquet vectorized reads when column types are promoted
+* [\#1991](https://github.com/apache/iceberg/pull/1991) fixes Avro schema conversions to preserve field docs
+* [\#1981](https://github.com/apache/iceberg/pull/1981) fixes date and timestamp transforms
Review comment:
I'd like to have better descriptions of the important bug fixes. Release notes should provide context for understanding what's in the release. In this case, date and timestamp transforms were producing incorrect values for negative dates and times. Being a bit more specific helps people understand that they are probably not affected. And if they are, they should click to the issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564876499
##########
File path: site/docs/releases.md
##########
@@ -51,44 +51,60 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`:
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-core</artifactId>
- <version>0.10.0</version>
+ <version>0.11.0</version>
</dependency>
...
</dependencies>
```
-## 0.10.0 release notes
+## 0.11.0 release notes
High-level features:
-* **Format v2 support** for building row-level operations (`MERGE INTO`) in processing engines
- * Note: format v2 is not yet finalized and does not have a forward-compatibility guarantee
-* **Flink integration** for writing to Iceberg tables and reading from Iceberg tables (reading supports batch mode only)
-* **Hive integration** for reading from Iceberg tables, with filter pushdown (experimental; configuration may change)
+* **Core API** now supports partition spec and sort order evolution
+* **Spark 3** now supports the following SQL extensions:
+ * MERGE INTO (experimental)
+ * DELETE FROM (experimental)
+ * ALTER TABLE ... ADD/DROP PARTITION
+ * ALTER TABLE ... WRITE ORDERED BY
+ * invoke stored procedures using CALL
+* **Flink** now supports streaming reads, CDC writes (experimental), and filter pushdown
+* **AWS module** is added to support better integration with AWS, with AWS Glue catalog support and dedicated S3 FileIO implementation
+* **Nessie module** is added to support integration with project Nessie
Important bug fixes:
-* [\#1706](https://github.com/apache/iceberg/pull/1706) fixes non-vectorized ORC reads in Spark that incorrectly skipped rows
-* [\#1536](https://github.com/apache/iceberg/pull/1536) fixes ORC conversion of `notIn` and `notEqual` to match null values
-* [\#1722](https://github.com/apache/iceberg/pull/1722) fixes `Expressions.notNull` returning an `isNull` predicate; API only, method was not used by processing engines
-* [\#1736](https://github.com/apache/iceberg/pull/1736) fixes `IllegalArgumentException` in vectorized Spark reads with negative decimal values
-* [\#1666](https://github.com/apache/iceberg/pull/1666) fixes file lengths returned by the ORC writer, using compressed size rather than uncompressed size
-* [\#1674](https://github.com/apache/iceberg/pull/1674) removes catalog expiration in HiveCatalogs
-* [\#1545](https://github.com/apache/iceberg/pull/1545) automatically refreshes tables in Spark when not caching table instances
+* [\#2091](https://github.com/apache/iceberg/pull/2091) fixes Parquet vectorized reads when column types are promoted
+* [\#1991](https://github.com/apache/iceberg/pull/1991) fixes Avro schema conversions to preserve field docs
+* [\#1981](https://github.com/apache/iceberg/pull/1981) fixes date and timestamp transforms
+* [\#1962](https://github.com/apache/iceberg/pull/1962) fixes Parquet vectorized position reader
+* [\#1811](https://github.com/apache/iceberg/pull/1811) makes refreshing Spark cache optional
+* [\#1798](https://github.com/apache/iceberg/pull/1798) fixes read failure when encountering duplicate entries of data files
+* [\#1785](https://github.com/apache/iceberg/pull/1785) fixes invalidation of metadata tables in CachingCatalog
+* [\#1784](https://github.com/apache/iceberg/pull/1784) fixes resolving of SparkSession table's metadata tables
Other notable changes:
-* The `iceberg-hive` module has been renamed to `iceberg-hive-metastore` to avoid confusion
-* Spark 3 is based on 3.0.1 that includes the fix for [SPARK-32168](https://issues.apache.org/jira/browse/SPARK-32168)
-* Hadoop tables will recover from version hint corruption
-* Tables can be configured with a required sort order
-* Data file locations can be customized with a dynamically loaded `LocationProvider`
-* ORC file imports can apply a name mapping for stats
+* NaN counter is added to format v2 metrics
+* Shared catalog properties are added in core library to standardize catalog level configurations
+* Spark and Flink now supports dynamically loading customized `Catalog` and `FileIO` implementations
+* Spark now supports loading tables with file paths via HadoopTables
+* Spark 2 now supports loading tables from other catalogs, like Spark 3
+* Spark 3 now supports catalog names in DataFrameReader when using Iceberg as a format
+* Hive now supports INSERT INTO, case insensitive query, projection pushdown, create DDL with schema and auto type conversion
+* ORC now supports reading tinyint, smallint, char, varchar types
+* Hadoop catalog now supports role-based access of table listing
+## Past releases
-A more exhaustive list of changes is available under the [0.10.0 release milestone](https://github.com/apache/iceberg/milestone/10?closed=1).
+### 0.10.0
Review comment:
yeah, I added it back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564861985
##########
File path: site/docs/releases.md
##########
@@ -51,44 +51,60 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`:
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-core</artifactId>
- <version>0.10.0</version>
+ <version>0.11.0</version>
</dependency>
...
</dependencies>
```
-## 0.10.0 release notes
+## 0.11.0 release notes
High-level features:
-* **Format v2 support** for building row-level operations (`MERGE INTO`) in processing engines
- * Note: format v2 is not yet finalized and does not have a forward-compatibility guarantee
-* **Flink integration** for writing to Iceberg tables and reading from Iceberg tables (reading supports batch mode only)
-* **Hive integration** for reading from Iceberg tables, with filter pushdown (experimental; configuration may change)
+* **Core API** now supports partition spec and sort order evolution
+* **Spark 3** now supports the following SQL extensions:
+ * MERGE INTO (experimental)
+ * DELETE FROM (experimental)
+ * ALTER TABLE ... ADD/DROP PARTITION
+ * ALTER TABLE ... WRITE ORDERED BY
+ * invoke stored procedures using CALL
+* **Flink** now supports streaming reads, CDC writes (experimental), and filter pushdown
+* **AWS module** is added to support better integration with AWS, with AWS Glue catalog support and dedicated S3 FileIO implementation
+* **Nessie module** is added to support integration with project Nessie
Important bug fixes:
-* [\#1706](https://github.com/apache/iceberg/pull/1706) fixes non-vectorized ORC reads in Spark that incorrectly skipped rows
-* [\#1536](https://github.com/apache/iceberg/pull/1536) fixes ORC conversion of `notIn` and `notEqual` to match null values
-* [\#1722](https://github.com/apache/iceberg/pull/1722) fixes `Expressions.notNull` returning an `isNull` predicate; API only, method was not used by processing engines
-* [\#1736](https://github.com/apache/iceberg/pull/1736) fixes `IllegalArgumentException` in vectorized Spark reads with negative decimal values
-* [\#1666](https://github.com/apache/iceberg/pull/1666) fixes file lengths returned by the ORC writer, using compressed size rather than uncompressed size
-* [\#1674](https://github.com/apache/iceberg/pull/1674) removes catalog expiration in HiveCatalogs
-* [\#1545](https://github.com/apache/iceberg/pull/1545) automatically refreshes tables in Spark when not caching table instances
+* [\#2091](https://github.com/apache/iceberg/pull/2091) fixes Parquet vectorized reads when column types are promoted
+* [\#1991](https://github.com/apache/iceberg/pull/1991) fixes Avro schema conversions to preserve field docs
+* [\#1981](https://github.com/apache/iceberg/pull/1981) fixes date and timestamp transforms
+* [\#1962](https://github.com/apache/iceberg/pull/1962) fixes Parquet vectorized position reader
+* [\#1811](https://github.com/apache/iceberg/pull/1811) makes refreshing Spark cache optional
+* [\#1798](https://github.com/apache/iceberg/pull/1798) fixes read failure when encountering duplicate entries of data files
+* [\#1785](https://github.com/apache/iceberg/pull/1785) fixes invalidation of metadata tables in CachingCatalog
+* [\#1784](https://github.com/apache/iceberg/pull/1784) fixes resolving of SparkSession table's metadata tables
Other notable changes:
-* The `iceberg-hive` module has been renamed to `iceberg-hive-metastore` to avoid confusion
-* Spark 3 is based on 3.0.1 that includes the fix for [SPARK-32168](https://issues.apache.org/jira/browse/SPARK-32168)
-* Hadoop tables will recover from version hint corruption
-* Tables can be configured with a required sort order
-* Data file locations can be customized with a dynamically loaded `LocationProvider`
-* ORC file imports can apply a name mapping for stats
+* NaN counter is added to format v2 metrics
+* Shared catalog properties are added in core library to standardize catalog level configurations
+* Spark and Flink now supports dynamically loading customized `Catalog` and `FileIO` implementations
+* Spark now supports loading tables with file paths via HadoopTables
Review comment:
This can be removed. It is only to support the existing behavior in `IcebergSource` now that the source supports multi-catalog table references.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564870592
##########
File path: site/docs/javadoc/index.html
##########
@@ -1,9 +1,9 @@
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Iceberg Javadoc Redirect</title>
- <meta http-equiv="refresh" content="0;URL='/javadoc/0.10.0/'" />
+ <meta http-equiv="refresh" content="0;URL='/javadoc/0.11.0/'" />
</head>
<body>
- <p>Redirecting to Javadoc for the 0.10.0 release: <a href="/javadoc/0.10.0/">/javadoc/0.10.0</a>.</p>
+ <p>Redirecting to Javadoc for the 0.11.0 release: <a href="/javadoc/0.11.0/">/javadoc/0.11.0</a>.</p>
Review comment:
Thanks, I added a note in README.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564863059
##########
File path: site/docs/releases.md
##########
@@ -51,44 +51,60 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`:
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-core</artifactId>
- <version>0.10.0</version>
+ <version>0.11.0</version>
</dependency>
...
</dependencies>
```
-## 0.10.0 release notes
+## 0.11.0 release notes
High-level features:
-* **Format v2 support** for building row-level operations (`MERGE INTO`) in processing engines
- * Note: format v2 is not yet finalized and does not have a forward-compatibility guarantee
-* **Flink integration** for writing to Iceberg tables and reading from Iceberg tables (reading supports batch mode only)
-* **Hive integration** for reading from Iceberg tables, with filter pushdown (experimental; configuration may change)
+* **Core API** now supports partition spec and sort order evolution
+* **Spark 3** now supports the following SQL extensions:
+ * MERGE INTO (experimental)
+ * DELETE FROM (experimental)
+ * ALTER TABLE ... ADD/DROP PARTITION
+ * ALTER TABLE ... WRITE ORDERED BY
+ * invoke stored procedures using CALL
+* **Flink** now supports streaming reads, CDC writes (experimental), and filter pushdown
+* **AWS module** is added to support better integration with AWS, with AWS Glue catalog support and dedicated S3 FileIO implementation
+* **Nessie module** is added to support integration with project Nessie
Important bug fixes:
-* [\#1706](https://github.com/apache/iceberg/pull/1706) fixes non-vectorized ORC reads in Spark that incorrectly skipped rows
-* [\#1536](https://github.com/apache/iceberg/pull/1536) fixes ORC conversion of `notIn` and `notEqual` to match null values
-* [\#1722](https://github.com/apache/iceberg/pull/1722) fixes `Expressions.notNull` returning an `isNull` predicate; API only, method was not used by processing engines
-* [\#1736](https://github.com/apache/iceberg/pull/1736) fixes `IllegalArgumentException` in vectorized Spark reads with negative decimal values
-* [\#1666](https://github.com/apache/iceberg/pull/1666) fixes file lengths returned by the ORC writer, using compressed size rather than uncompressed size
-* [\#1674](https://github.com/apache/iceberg/pull/1674) removes catalog expiration in HiveCatalogs
-* [\#1545](https://github.com/apache/iceberg/pull/1545) automatically refreshes tables in Spark when not caching table instances
+* [\#2091](https://github.com/apache/iceberg/pull/2091) fixes Parquet vectorized reads when column types are promoted
+* [\#1991](https://github.com/apache/iceberg/pull/1991) fixes Avro schema conversions to preserve field docs
+* [\#1981](https://github.com/apache/iceberg/pull/1981) fixes date and timestamp transforms
+* [\#1962](https://github.com/apache/iceberg/pull/1962) fixes Parquet vectorized position reader
+* [\#1811](https://github.com/apache/iceberg/pull/1811) makes refreshing Spark cache optional
+* [\#1798](https://github.com/apache/iceberg/pull/1798) fixes read failure when encountering duplicate entries of data files
+* [\#1785](https://github.com/apache/iceberg/pull/1785) fixes invalidation of metadata tables in CachingCatalog
+* [\#1784](https://github.com/apache/iceberg/pull/1784) fixes resolving of SparkSession table's metadata tables
Other notable changes:
-* The `iceberg-hive` module has been renamed to `iceberg-hive-metastore` to avoid confusion
-* Spark 3 is based on 3.0.1 that includes the fix for [SPARK-32168](https://issues.apache.org/jira/browse/SPARK-32168)
-* Hadoop tables will recover from version hint corruption
-* Tables can be configured with a required sort order
-* Data file locations can be customized with a dynamically loaded `LocationProvider`
-* ORC file imports can apply a name mapping for stats
+* NaN counter is added to format v2 metrics
+* Shared catalog properties are added in core library to standardize catalog level configurations
+* Spark and Flink now supports dynamically loading customized `Catalog` and `FileIO` implementations
+* Spark now supports loading tables with file paths via HadoopTables
+* Spark 2 now supports loading tables from other catalogs, like Spark 3
+* Spark 3 now supports catalog names in DataFrameReader when using Iceberg as a format
+* Hive now supports INSERT INTO, case insensitive query, projection pushdown, create DDL with schema and auto type conversion
+* ORC now supports reading tinyint, smallint, char, varchar types
+* Hadoop catalog now supports role-based access of table listing
Review comment:
This can be removed because it is misleading. Role-based access isn't what was supported. The fix was to avoid failing to list when some tables cannot be read due to permissions errors. Instead of failing, the tables are ignored.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564860582
##########
File path: site/docs/releases.md
##########
@@ -51,44 +51,60 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`:
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-core</artifactId>
- <version>0.10.0</version>
+ <version>0.11.0</version>
</dependency>
...
</dependencies>
```
-## 0.10.0 release notes
+## 0.11.0 release notes
High-level features:
-* **Format v2 support** for building row-level operations (`MERGE INTO`) in processing engines
- * Note: format v2 is not yet finalized and does not have a forward-compatibility guarantee
-* **Flink integration** for writing to Iceberg tables and reading from Iceberg tables (reading supports batch mode only)
-* **Hive integration** for reading from Iceberg tables, with filter pushdown (experimental; configuration may change)
+* **Core API** now supports partition spec and sort order evolution
+* **Spark 3** now supports the following SQL extensions:
+ * MERGE INTO (experimental)
+ * DELETE FROM (experimental)
+ * ALTER TABLE ... ADD/DROP PARTITION
+ * ALTER TABLE ... WRITE ORDERED BY
+ * invoke stored procedures using CALL
+* **Flink** now supports streaming reads, CDC writes (experimental), and filter pushdown
+* **AWS module** is added to support better integration with AWS, with AWS Glue catalog support and dedicated S3 FileIO implementation
+* **Nessie module** is added to support integration with project Nessie
Important bug fixes:
-* [\#1706](https://github.com/apache/iceberg/pull/1706) fixes non-vectorized ORC reads in Spark that incorrectly skipped rows
-* [\#1536](https://github.com/apache/iceberg/pull/1536) fixes ORC conversion of `notIn` and `notEqual` to match null values
-* [\#1722](https://github.com/apache/iceberg/pull/1722) fixes `Expressions.notNull` returning an `isNull` predicate; API only, method was not used by processing engines
-* [\#1736](https://github.com/apache/iceberg/pull/1736) fixes `IllegalArgumentException` in vectorized Spark reads with negative decimal values
-* [\#1666](https://github.com/apache/iceberg/pull/1666) fixes file lengths returned by the ORC writer, using compressed size rather than uncompressed size
-* [\#1674](https://github.com/apache/iceberg/pull/1674) removes catalog expiration in HiveCatalogs
-* [\#1545](https://github.com/apache/iceberg/pull/1545) automatically refreshes tables in Spark when not caching table instances
+* [\#2091](https://github.com/apache/iceberg/pull/2091) fixes Parquet vectorized reads when column types are promoted
+* [\#1991](https://github.com/apache/iceberg/pull/1991) fixes Avro schema conversions to preserve field docs
+* [\#1981](https://github.com/apache/iceberg/pull/1981) fixes date and timestamp transforms
+* [\#1962](https://github.com/apache/iceberg/pull/1962) fixes Parquet vectorized position reader
+* [\#1811](https://github.com/apache/iceberg/pull/1811) makes refreshing Spark cache optional
+* [\#1798](https://github.com/apache/iceberg/pull/1798) fixes read failure when encountering duplicate entries of data files
+* [\#1785](https://github.com/apache/iceberg/pull/1785) fixes invalidation of metadata tables in CachingCatalog
+* [\#1784](https://github.com/apache/iceberg/pull/1784) fixes resolving of SparkSession table's metadata tables
Review comment:
This is also internal and probably not worth mentioning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564860874
##########
File path: site/docs/releases.md
##########
@@ -51,44 +51,60 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`:
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-core</artifactId>
- <version>0.10.0</version>
+ <version>0.11.0</version>
</dependency>
...
</dependencies>
```
-## 0.10.0 release notes
+## 0.11.0 release notes
High-level features:
-* **Format v2 support** for building row-level operations (`MERGE INTO`) in processing engines
- * Note: format v2 is not yet finalized and does not have a forward-compatibility guarantee
-* **Flink integration** for writing to Iceberg tables and reading from Iceberg tables (reading supports batch mode only)
-* **Hive integration** for reading from Iceberg tables, with filter pushdown (experimental; configuration may change)
+* **Core API** now supports partition spec and sort order evolution
+* **Spark 3** now supports the following SQL extensions:
+ * MERGE INTO (experimental)
+ * DELETE FROM (experimental)
+ * ALTER TABLE ... ADD/DROP PARTITION
+ * ALTER TABLE ... WRITE ORDERED BY
+ * invoke stored procedures using CALL
+* **Flink** now supports streaming reads, CDC writes (experimental), and filter pushdown
+* **AWS module** is added to support better integration with AWS, with AWS Glue catalog support and dedicated S3 FileIO implementation
+* **Nessie module** is added to support integration with project Nessie
Important bug fixes:
-* [\#1706](https://github.com/apache/iceberg/pull/1706) fixes non-vectorized ORC reads in Spark that incorrectly skipped rows
-* [\#1536](https://github.com/apache/iceberg/pull/1536) fixes ORC conversion of `notIn` and `notEqual` to match null values
-* [\#1722](https://github.com/apache/iceberg/pull/1722) fixes `Expressions.notNull` returning an `isNull` predicate; API only, method was not used by processing engines
-* [\#1736](https://github.com/apache/iceberg/pull/1736) fixes `IllegalArgumentException` in vectorized Spark reads with negative decimal values
-* [\#1666](https://github.com/apache/iceberg/pull/1666) fixes file lengths returned by the ORC writer, using compressed size rather than uncompressed size
-* [\#1674](https://github.com/apache/iceberg/pull/1674) removes catalog expiration in HiveCatalogs
-* [\#1545](https://github.com/apache/iceberg/pull/1545) automatically refreshes tables in Spark when not caching table instances
+* [\#2091](https://github.com/apache/iceberg/pull/2091) fixes Parquet vectorized reads when column types are promoted
+* [\#1991](https://github.com/apache/iceberg/pull/1991) fixes Avro schema conversions to preserve field docs
+* [\#1981](https://github.com/apache/iceberg/pull/1981) fixes date and timestamp transforms
+* [\#1962](https://github.com/apache/iceberg/pull/1962) fixes Parquet vectorized position reader
+* [\#1811](https://github.com/apache/iceberg/pull/1811) makes refreshing Spark cache optional
+* [\#1798](https://github.com/apache/iceberg/pull/1798) fixes read failure when encountering duplicate entries of data files
+* [\#1785](https://github.com/apache/iceberg/pull/1785) fixes invalidation of metadata tables in CachingCatalog
+* [\#1784](https://github.com/apache/iceberg/pull/1784) fixes resolving of SparkSession table's metadata tables
Other notable changes:
-* The `iceberg-hive` module has been renamed to `iceberg-hive-metastore` to avoid confusion
-* Spark 3 is based on 3.0.1 that includes the fix for [SPARK-32168](https://issues.apache.org/jira/browse/SPARK-32168)
-* Hadoop tables will recover from version hint corruption
-* Tables can be configured with a required sort order
-* Data file locations can be customized with a dynamically loaded `LocationProvider`
-* ORC file imports can apply a name mapping for stats
+* NaN counter is added to format v2 metrics
Review comment:
This can just state that NaN counts are now supported in metadata. It is optional in v1, required in v2 so we don't need to say that it was added to v2.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564858269
##########
File path: site/docs/javadoc/index.html
##########
@@ -1,9 +1,9 @@
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Iceberg Javadoc Redirect</title>
- <meta http-equiv="refresh" content="0;URL='/javadoc/0.10.0/'" />
+ <meta http-equiv="refresh" content="0;URL='/javadoc/0.11.0/'" />
</head>
<body>
- <p>Redirecting to Javadoc for the 0.10.0 release: <a href="/javadoc/0.10.0/">/javadoc/0.10.0</a>.</p>
+ <p>Redirecting to Javadoc for the 0.11.0 release: <a href="/javadoc/0.11.0/">/javadoc/0.11.0</a>.</p>
Review comment:
It's in `tasks.gradle`. Run `./gradlew refreshJavadoc`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2157: Doc: update 0.11 release notes
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2157:
URL: https://github.com/apache/iceberg/pull/2157#discussion_r564859480
##########
File path: site/docs/releases.md
##########
@@ -51,44 +51,60 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`:
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-core</artifactId>
- <version>0.10.0</version>
+ <version>0.11.0</version>
</dependency>
...
</dependencies>
```
-## 0.10.0 release notes
+## 0.11.0 release notes
High-level features:
-* **Format v2 support** for building row-level operations (`MERGE INTO`) in processing engines
- * Note: format v2 is not yet finalized and does not have a forward-compatibility guarantee
-* **Flink integration** for writing to Iceberg tables and reading from Iceberg tables (reading supports batch mode only)
-* **Hive integration** for reading from Iceberg tables, with filter pushdown (experimental; configuration may change)
+* **Core API** now supports partition spec and sort order evolution
+* **Spark 3** now supports the following SQL extensions:
+ * MERGE INTO (experimental)
+ * DELETE FROM (experimental)
+ * ALTER TABLE ... ADD/DROP PARTITION
+ * ALTER TABLE ... WRITE ORDERED BY
+ * invoke stored procedures using CALL
+* **Flink** now supports streaming reads, CDC writes (experimental), and filter pushdown
+* **AWS module** is added to support better integration with AWS, with AWS Glue catalog support and dedicated S3 FileIO implementation
+* **Nessie module** is added to support integration with project Nessie
Important bug fixes:
-* [\#1706](https://github.com/apache/iceberg/pull/1706) fixes non-vectorized ORC reads in Spark that incorrectly skipped rows
-* [\#1536](https://github.com/apache/iceberg/pull/1536) fixes ORC conversion of `notIn` and `notEqual` to match null values
-* [\#1722](https://github.com/apache/iceberg/pull/1722) fixes `Expressions.notNull` returning an `isNull` predicate; API only, method was not used by processing engines
-* [\#1736](https://github.com/apache/iceberg/pull/1736) fixes `IllegalArgumentException` in vectorized Spark reads with negative decimal values
-* [\#1666](https://github.com/apache/iceberg/pull/1666) fixes file lengths returned by the ORC writer, using compressed size rather than uncompressed size
-* [\#1674](https://github.com/apache/iceberg/pull/1674) removes catalog expiration in HiveCatalogs
-* [\#1545](https://github.com/apache/iceberg/pull/1545) automatically refreshes tables in Spark when not caching table instances
+* [\#2091](https://github.com/apache/iceberg/pull/2091) fixes Parquet vectorized reads when column types are promoted
+* [\#1991](https://github.com/apache/iceberg/pull/1991) fixes Avro schema conversions to preserve field docs
+* [\#1981](https://github.com/apache/iceberg/pull/1981) fixes date and timestamp transforms
+* [\#1962](https://github.com/apache/iceberg/pull/1962) fixes Parquet vectorized position reader
+* [\#1811](https://github.com/apache/iceberg/pull/1811) makes refreshing Spark cache optional
+* [\#1798](https://github.com/apache/iceberg/pull/1798) fixes read failure when encountering duplicate entries of data files
+* [\#1785](https://github.com/apache/iceberg/pull/1785) fixes invalidation of metadata tables in CachingCatalog
+* [\#1784](https://github.com/apache/iceberg/pull/1784) fixes resolving of SparkSession table's metadata tables
Other notable changes:
-* The `iceberg-hive` module has been renamed to `iceberg-hive-metastore` to avoid confusion
-* Spark 3 is based on 3.0.1 that includes the fix for [SPARK-32168](https://issues.apache.org/jira/browse/SPARK-32168)
-* Hadoop tables will recover from version hint corruption
-* Tables can be configured with a required sort order
-* Data file locations can be customized with a dynamically loaded `LocationProvider`
-* ORC file imports can apply a name mapping for stats
+* NaN counter is added to format v2 metrics
+* Shared catalog properties are added in core library to standardize catalog level configurations
+* Spark and Flink now supports dynamically loading customized `Catalog` and `FileIO` implementations
+* Spark now supports loading tables with file paths via HadoopTables
+* Spark 2 now supports loading tables from other catalogs, like Spark 3
+* Spark 3 now supports catalog names in DataFrameReader when using Iceberg as a format
+* Hive now supports INSERT INTO, case insensitive query, projection pushdown, create DDL with schema and auto type conversion
Review comment:
Can you change this to `Hive (experimental) ...)`? There are a few bugs with the insert feature, and we haven't finished fixing how table references are stored.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org