You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/05/31 11:47:57 UTC
[GitHub] [iceberg] pvary opened a new pull request, #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
pvary opened a new pull request, #4915:
URL: https://github.com/apache/iceberg/pull/4915
Update Hive doc page with the 4.0.0-alpha-1 features
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1170251504
Looks good to me. Can you rebase? I don't think that we need the _index.md file any more. @samredai, how should we update this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#discussion_r886649585
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting data by appending it to a table (MapReduce only)
Review Comment:
Done
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting data by appending it to a table (MapReduce only)
+
+Iceberg compatibility with Hive 4.0.0-alpha-1
+(Using HiveCatalog) supports the following, additional features:
+
+* Creating an Iceberg identity-partitioned table
+* Creating an Iceberg identity-partitioned table from a spec
+* Creating a table from an existing table (CTAS table)
+* Altering a table while keeping Iceberg and Hive schemas in sync
+* Altering the partition schema (updating columns)
+* Altering the partition schema by specifying partition transforms
+* Truncating a table
+* Migrating tables in Avro, Parquet, or ORC format to Iceberg
+* Reading the schema of a table
+* Time travel applications
+* Inserting data by appending it into a table (Tez only)
+* Inserting data overwriting existing data
Review Comment:
Done
##########
docs/hive/_index.md:
##########
@@ -234,89 +337,118 @@ to the shared data files in case of accidental drop table commands from the Hive
which would unintentionally remove all the data in the table.
{{< /hint >}}
-### DROP TABLE
-
-Tables can be dropped using the `DROP TABLE` command:
+### ALTER TABLE
+#### Table properties
+For HiveCatalog tables the Iceberg table properties and the Hive table properties stored in HMS are kept in sync.
+{{< hint info >}}
+IMPORTANT: This feature is not available for other Catalog implementations.
+{{< /hint >}}
```sql
-DROP TABLE [IF EXISTS] table_name [PURGE];
+ALTER TABLE t SET TBLPROPERTIES('...'='...');
```
-You can configure purge behavior through global Hadoop configuration or Hive metastore table properties:
-
-| Config key | Default | Description |
-| ----------------------------| ---------------------------| --------------------------------------------------------------- |
-| external.table.purge | true | if all data and metadata should be purged in a table by default |
-
-Each Iceberg table's default purge behavior can also be configured through Iceberg table properties:
-
-| Property | Default | Description |
-| ----------------------------| ---------------------------| ----------------------------------------------------------------- |
-| gc.enabled | true | if all data and metadata should be purged in the table by default |
+#### Schema evolution
+The Hive table schema is kept in sync with the Iceberg table. If an outside source (Impala/Spark/Java API/etc)
+changes the schema, the Hive table immediately reflects the changes. You alter the table schema using Hive commands:
-When changing `gc.enabled` on the Iceberg table via `UpdateProperties`, `external.table.purge` is also updated on HMS table accordingly.
-When setting `external.table.purge` as a table prop during Hive `CREATE TABLE`, `gc.enabled` is pushed down accordingly to the Iceberg table properties.
-This makes sure that the 2 properties are always consistent at table level between Hive and Iceberg.
-
-{{< hint danger >}}
-Changing `external.table.purge` via Hive `ALTER TABLE SET TBLPROPERTIES` does not update `gc.enabled` on the Iceberg table.
-This is a limitation on Hive 3.1.2 because the `HiveMetaHook` doesn't have all the hooks for alter tables yet.
-{{< /hint >}}
-
-## Querying with SQL
-
-Here are the features highlights for Iceberg Hive read support:
-
-1. **Predicate pushdown**: Pushdown of the Hive SQL `WHERE` clause has been implemented so that these filters are used at the Iceberg `TableScan` level as well as by the Parquet and ORC Readers.
-2. **Column projection**: Columns from the Hive SQL `SELECT` clause are projected down to the Iceberg readers to reduce the number of columns read.
-3. **Hive query engines**: Both the MapReduce and Tez query execution engines are supported.
-
-### Configurations
-
-Here are the Hadoop configurations that one can adjust for the Hive reader:
+* Add a column
+```sql
+ALTER TABLE orders ADD COLUMNS (nickname string);
+```
+* Rename a column
+```sql
+ALTER TABLE orders CHANGE COLUMN item fruit string;
+```
+* Reorder columns
+```sql
+ALTER TABLE orders CHANGE COLUMN quantity quantity int AFTER price;
+```
+* Change a column type - only if the Iceberg defined the column type change as safe
+```sql
+ALTER TABLE orders CHANGE COLUMN price price long;
+```
+* Drop column by using CHANGE COLUMN to remove the old column
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
samredai commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1168722355
@pvary I opened a [PR](https://github.com/apache/iceberg-docs/pull/98) with these changes directly against the `latest` branch and we'll need to do the same for the `0.13.2` branch. That way this can get released with the current format of the docs site and I'll make sure those changes are carried over to the new site format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1168724370
Thanks @samredai!
Are you eligible to +1/merge, or shall we wait for @rdblue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
samredai commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1170377953
Ah that's right, this has already been added by PR #5115 . That includes all of the changes from this PR here so this should be good to close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a diff in pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
samredai commented on code in PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#discussion_r889493560
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
Review Comment:
These versions still support this so it should be present tense, right? So `s/supported/supports`
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting into a table (will append rows) (MapReduce only)
Review Comment:
Can `(will append rows)` and `(MapReduce only)` be expanded to warning/info boxes? Something along the lines of:
```
{{< hint info >}}
Inserting into a table will append rows to the relevant partitions.
{{< /hint >}}
{{< hint warning >}}
Inserting into a table works with MapReduce only.
{{< /hint >}}
```
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting into a table (will append rows) (MapReduce only)
+
+Iceberg compatibility with Hive 4.0.0-alpha-1
+(Using HiveCatalog) supports the following, additional features:
+
+* Creating an Iceberg identity-partitioned table
+* Creating an Iceberg table with any partition spec (including the various transforms Iceberg supports)
Review Comment:
I don't think the parentheses are needed here. I'd suggest:
```
* Creating an Iceberg table with any partition spec, including the various transforms supported by Iceberg
```
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting into a table (will append rows) (MapReduce only)
+
+Iceberg compatibility with Hive 4.0.0-alpha-1
+(Using HiveCatalog) supports the following, additional features:
Review Comment:
nit: I'd word this as
> With Hive version 4.0.0-alpha-1 and above, the Iceberg `HiveCatalog` supports the following additional features:
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting into a table (will append rows) (MapReduce only)
+
+Iceberg compatibility with Hive 4.0.0-alpha-1
+(Using HiveCatalog) supports the following, additional features:
+
+* Creating an Iceberg identity-partitioned table
+* Creating an Iceberg table with any partition spec (including the various transforms Iceberg supports)
+* Creating a table from an existing table (CTAS table)
+* Altering a table while keeping Iceberg and Hive schemas in sync
+* Altering the partition schema (updating columns)
+* Altering the partition schema by specifying partition transforms
+* Truncating a table
+* Migrating tables in Avro, Parquet, or ORC (Non-ACID) format to Iceberg
+* Reading the schema of a table
+* Time travel applications
+* Inserting into a table (will append rows) (Tez only)
Review Comment:
Similar to my other comment, I would put these in warning or info boxes instead of tacking them onto the end of the line. Those boxes are more visible disclaimers.
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting into a table (will append rows) (MapReduce only)
+
+Iceberg compatibility with Hive 4.0.0-alpha-1
+(Using HiveCatalog) supports the following, additional features:
+
+* Creating an Iceberg identity-partitioned table
+* Creating an Iceberg table with any partition spec (including the various transforms Iceberg supports)
+* Creating a table from an existing table (CTAS table)
+* Altering a table while keeping Iceberg and Hive schemas in sync
+* Altering the partition schema (updating columns)
+* Altering the partition schema by specifying partition transforms
+* Truncating a table
+* Migrating tables in Avro, Parquet, or ORC (Non-ACID) format to Iceberg
+* Reading the schema of a table
+* Time travel applications
+* Inserting into a table (will append rows) (Tez only)
+* Inserting data overwriting existing data
## Enabling Iceberg support in Hive
-### Loading runtime jar
+### Hive 4.0.0-alpha-1
+
+Hive 4.0.0-alpha-1 comes with the Iceberg 0.13.1 included, no additional downloads or jars are needed.
Review Comment:
```suggestion
Hive 4.0.0-alpha-1 comes with the Iceberg 0.13.1 included. No additional downloads or jars are needed.
```
##########
docs/hive/_index.md:
##########
@@ -65,7 +90,7 @@ For example, setting this in the `hive-site.xml` loaded by Spark will enable the
Starting with Apache Iceberg `0.11.0`, when using Hive with Tez you also have to disable vectorization (`hive.vectorized.execution.enabled=false`).
{{< /hint >}}
-#### Table property configuration
+##### Table property configuration
Review Comment:
Keep in mind these changes may impact what shows up in the table of contents for this page. IIRC, h5 headers don't show up in the TOC.
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting into a table (will append rows) (MapReduce only)
+
+Iceberg compatibility with Hive 4.0.0-alpha-1
+(Using HiveCatalog) supports the following, additional features:
+
+* Creating an Iceberg identity-partitioned table
+* Creating an Iceberg table with any partition spec (including the various transforms Iceberg supports)
+* Creating a table from an existing table (CTAS table)
+* Altering a table while keeping Iceberg and Hive schemas in sync
+* Altering the partition schema (updating columns)
+* Altering the partition schema by specifying partition transforms
+* Truncating a table
+* Migrating tables in Avro, Parquet, or ORC (Non-ACID) format to Iceberg
+* Reading the schema of a table
+* Time travel applications
+* Inserting into a table (will append rows) (Tez only)
+* Inserting data overwriting existing data
## Enabling Iceberg support in Hive
-### Loading runtime jar
+### Hive 4.0.0-alpha-1
+
+Hive 4.0.0-alpha-1 comes with the Iceberg 0.13.1 included, no additional downloads or jars are needed.
+
+### Hive 2.3.x, Hive 3.1.x
+
+You need to do the following steps:
+* Loading runtime jar
+* Enabling support
Review Comment:
This could just be one line right?
> In order to use Hive 2.3.x or Hive 3.1.x, you must load the Iceberg-Hive runtime jar and enable Iceberg support, either globally or for an individual table using a table property.
and then jump into the sections below that provide more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lcspinter commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
lcspinter commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1144497213
Thanks @pvary for your work!
+1 non-binding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1169524313
Thanks @samredai for the help. With Ryan's review, I was able to merge https://github.com/apache/iceberg-docs/pull/98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#discussion_r890962377
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting into a table (will append rows) (MapReduce only)
+
+Iceberg compatibility with Hive 4.0.0-alpha-1
+(Using HiveCatalog) supports the following, additional features:
+
+* Creating an Iceberg identity-partitioned table
+* Creating an Iceberg table with any partition spec (including the various transforms Iceberg supports)
Review Comment:
Done
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting into a table (will append rows) (MapReduce only)
+
+Iceberg compatibility with Hive 4.0.0-alpha-1
+(Using HiveCatalog) supports the following, additional features:
+
+* Creating an Iceberg identity-partitioned table
+* Creating an Iceberg table with any partition spec (including the various transforms Iceberg supports)
+* Creating a table from an existing table (CTAS table)
+* Altering a table while keeping Iceberg and Hive schemas in sync
+* Altering the partition schema (updating columns)
+* Altering the partition schema by specifying partition transforms
+* Truncating a table
+* Migrating tables in Avro, Parquet, or ORC (Non-ACID) format to Iceberg
+* Reading the schema of a table
+* Time travel applications
+* Inserting into a table (will append rows) (Tez only)
+* Inserting data overwriting existing data
## Enabling Iceberg support in Hive
-### Loading runtime jar
+### Hive 4.0.0-alpha-1
+
+Hive 4.0.0-alpha-1 comes with the Iceberg 0.13.1 included, no additional downloads or jars are needed.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#discussion_r890962076
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting into a table (will append rows) (MapReduce only)
+
+Iceberg compatibility with Hive 4.0.0-alpha-1
+(Using HiveCatalog) supports the following, additional features:
Review Comment:
Reworded to this:
```
With Hive version 4.0.0-alpha-1 and above,
the Iceberg integration when using HiveCatalog supports the following additional features:
```
The point here is that some of the features are dependent on HiveCatalog, but HiveCatalog is not the full integration. You can use a different catalog implementation (say HadoopCatalog), and most of the features will work, but for example the Hive schema / Iceberg schema sync will work only with HiveCatalog.
Is this better? Any other suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1149822422
@samredai: Could you please check again? Went through your comments, and added 2 new paragraphs.
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#discussion_r886649378
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting data by appending it to a table (MapReduce only)
+
+Iceberg compatibility with Hive 4.0.0-alpha-1
+(Using HiveCatalog) supports the following, additional features:
+
+* Creating an Iceberg identity-partitioned table
+* Creating an Iceberg identity-partitioned table from a spec
Review Comment:
Done
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting data by appending it to a table (MapReduce only)
+
+Iceberg compatibility with Hive 4.0.0-alpha-1
+(Using HiveCatalog) supports the following, additional features:
+
+* Creating an Iceberg identity-partitioned table
+* Creating an Iceberg identity-partitioned table from a spec
+* Creating a table from an existing table (CTAS table)
+* Altering a table while keeping Iceberg and Hive schemas in sync
+* Altering the partition schema (updating columns)
+* Altering the partition schema by specifying partition transforms
+* Truncating a table
+* Migrating tables in Avro, Parquet, or ORC format to Iceberg
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1143428354
> Generally looks good, some minor remarks.
Thanks @szlta for the review!
Could you please check the changes?
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#discussion_r890955688
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting into a table (will append rows) (MapReduce only)
Review Comment:
The first parenthesis was trying to highlight the difference between the `INSERT INTO` and the `INSERT OVERWRITE`.
Could you please check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
samredai commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1170322383
Instead of `_index.md` it should be renamed to `hive.md` and moved up one directory, which means the `hive` directory can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
samredai commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1170395879
Here's a rebased PR #5161 with the content added to the existing `hive.md` file. @pvary can you confirm the content all looks good? Then we can merge that and close this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1170398235
> Here's a rebased PR #5161 with the content added to the existing `hive.md` file. @pvary can you confirm the content all looks good? Then we can merge that and close this PR.
Will check it tomorrow. It is quite late here, and the change is big.
Thanks for all the help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary closed pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary closed pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
URL: https://github.com/apache/iceberg/pull/4915
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#discussion_r890962780
##########
docs/hive/_index.md:
##########
@@ -65,7 +90,7 @@ For example, setting this in the `hive-site.xml` loaded by Spark will enable the
Starting with Apache Iceberg `0.11.0`, when using Hive with Tez you also have to disable vectorization (`hive.vectorized.execution.enabled=false`).
{{< /hint >}}
-#### Table property configuration
+##### Table property configuration
Review Comment:
I think this is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szlta commented on a diff in pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
szlta commented on code in PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#discussion_r886565431
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting data by appending it to a table (MapReduce only)
Review Comment:
nit: sounds bit strange to me, perhaps:
- Inserting data into a table (will append rows) (MapReduce only)
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting data by appending it to a table (MapReduce only)
+
+Iceberg compatibility with Hive 4.0.0-alpha-1
+(Using HiveCatalog) supports the following, additional features:
+
+* Creating an Iceberg identity-partitioned table
+* Creating an Iceberg identity-partitioned table from a spec
Review Comment:
Maybe:
- Creating an Iceberg table with any partition spec (including the various transforms Iceberg supports)
..as I think the feature is there in alpha1: https://github.com/apache/hive/commit/eedcd82bc2d61861a27205f925ba0ffab9b6bca8
also, see "Partitioned tables" part
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting data by appending it to a table (MapReduce only)
+
+Iceberg compatibility with Hive 4.0.0-alpha-1
+(Using HiveCatalog) supports the following, additional features:
+
+* Creating an Iceberg identity-partitioned table
+* Creating an Iceberg identity-partitioned table from a spec
+* Creating a table from an existing table (CTAS table)
+* Altering a table while keeping Iceberg and Hive schemas in sync
+* Altering the partition schema (updating columns)
+* Altering the partition schema by specifying partition transforms
+* Truncating a table
+* Migrating tables in Avro, Parquet, or ORC format to Iceberg
+* Reading the schema of a table
+* Time travel applications
+* Inserting data by appending it into a table (Tez only)
+* Inserting data overwriting existing data
Review Comment:
nit: similarly:
- Inserting into a table (will append rows) (Tez only)
- Insert overrite an existing table
##########
docs/hive/_index.md:
##########
@@ -24,19 +24,44 @@ weight: 400
# Hive
Iceberg supports reading and writing Iceberg tables through [Hive](https://hive.apache.org) by using a [StorageHandler](https://cwiki.apache.org/confluence/display/Hive/StorageHandlers).
-Here is the current compatibility matrix for Iceberg Hive support:
-| Feature | Hive 2.x | Hive 3.1.2 |
-| ------------------------ | ---------------------- | ---------------------- |
-| CREATE EXTERNAL TABLE | ✔️ | ✔️ |
-| CREATE TABLE | ✔️ | ✔️ |
-| DROP TABLE | ✔️ | ✔️ |
-| SELECT | ✔️ (MapReduce and Tez) | ✔️ (MapReduce and Tez) |
-| INSERT INTO | ✔️ (MapReduce only)️ | ✔️ (MapReduce only) |
+## Feature support
+Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supported the following features:
+
+* Creating a table
+* Dropping a table
+* Reading a table
+* Inserting data by appending it to a table (MapReduce only)
+
+Iceberg compatibility with Hive 4.0.0-alpha-1
+(Using HiveCatalog) supports the following, additional features:
+
+* Creating an Iceberg identity-partitioned table
+* Creating an Iceberg identity-partitioned table from a spec
+* Creating a table from an existing table (CTAS table)
+* Altering a table while keeping Iceberg and Hive schemas in sync
+* Altering the partition schema (updating columns)
+* Altering the partition schema by specifying partition transforms
+* Truncating a table
+* Migrating tables in Avro, Parquet, or ORC format to Iceberg
Review Comment:
Should we point out in ORC's case that migration is supported for only non-ACID ORC tables?
##########
docs/hive/_index.md:
##########
@@ -234,89 +337,118 @@ to the shared data files in case of accidental drop table commands from the Hive
which would unintentionally remove all the data in the table.
{{< /hint >}}
-### DROP TABLE
-
-Tables can be dropped using the `DROP TABLE` command:
+### ALTER TABLE
+#### Table properties
+For HiveCatalog tables the Iceberg table properties and the Hive table properties stored in HMS are kept in sync.
+{{< hint info >}}
+IMPORTANT: This feature is not available for other Catalog implementations.
+{{< /hint >}}
```sql
-DROP TABLE [IF EXISTS] table_name [PURGE];
+ALTER TABLE t SET TBLPROPERTIES('...'='...');
```
-You can configure purge behavior through global Hadoop configuration or Hive metastore table properties:
-
-| Config key | Default | Description |
-| ----------------------------| ---------------------------| --------------------------------------------------------------- |
-| external.table.purge | true | if all data and metadata should be purged in a table by default |
-
-Each Iceberg table's default purge behavior can also be configured through Iceberg table properties:
-
-| Property | Default | Description |
-| ----------------------------| ---------------------------| ----------------------------------------------------------------- |
-| gc.enabled | true | if all data and metadata should be purged in the table by default |
+#### Schema evolution
+The Hive table schema is kept in sync with the Iceberg table. If an outside source (Impala/Spark/Java API/etc)
+changes the schema, the Hive table immediately reflects the changes. You alter the table schema using Hive commands:
-When changing `gc.enabled` on the Iceberg table via `UpdateProperties`, `external.table.purge` is also updated on HMS table accordingly.
-When setting `external.table.purge` as a table prop during Hive `CREATE TABLE`, `gc.enabled` is pushed down accordingly to the Iceberg table properties.
-This makes sure that the 2 properties are always consistent at table level between Hive and Iceberg.
-
-{{< hint danger >}}
-Changing `external.table.purge` via Hive `ALTER TABLE SET TBLPROPERTIES` does not update `gc.enabled` on the Iceberg table.
-This is a limitation on Hive 3.1.2 because the `HiveMetaHook` doesn't have all the hooks for alter tables yet.
-{{< /hint >}}
-
-## Querying with SQL
-
-Here are the features highlights for Iceberg Hive read support:
-
-1. **Predicate pushdown**: Pushdown of the Hive SQL `WHERE` clause has been implemented so that these filters are used at the Iceberg `TableScan` level as well as by the Parquet and ORC Readers.
-2. **Column projection**: Columns from the Hive SQL `SELECT` clause are projected down to the Iceberg readers to reduce the number of columns read.
-3. **Hive query engines**: Both the MapReduce and Tez query execution engines are supported.
-
-### Configurations
-
-Here are the Hadoop configurations that one can adjust for the Hive reader:
+* Add a column
+```sql
+ALTER TABLE orders ADD COLUMNS (nickname string);
+```
+* Rename a column
+```sql
+ALTER TABLE orders CHANGE COLUMN item fruit string;
+```
+* Reorder columns
+```sql
+ALTER TABLE orders CHANGE COLUMN quantity quantity int AFTER price;
+```
+* Change a column type - only if the Iceberg defined the column type change as safe
+```sql
+ALTER TABLE orders CHANGE COLUMN price price long;
+```
+* Drop column by using CHANGE COLUMN to remove the old column
Review Comment:
not CHANGE COLUMN, it's REPLACE COLUMNS
Also note, that dropping columns is only thing REPLACE COLUMNS can be used for i.e. if columns are specified out-of-order an error will be thrown signalling this limitation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1170373166
@samredai: Any other change that we have to keep from the changes came in for docs/hive/_index.md? What I see are only formatting changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1170993443
Merged #5161 to master.
Thanks @samredai and @krishahn for the help!
Thanks @szlta, @lcspinter and @rdblue for the reviews!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1168708172
@samredai: I have checked the conflicts caused by #5115.
This seems to me the relevant part:
```
---
title: "Hive"
url: hive
weight: 400
menu: main
---
```
What about the other changes?
It would be good to merge the Hive integration part to the docs as well. What is needed for getting them in?
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on pull request #4915: [DOC] Update Hive doc page with the 4.0.0-alpha-1 features
Posted by GitBox <gi...@apache.org>.
samredai commented on PR #4915:
URL: https://github.com/apache/iceberg/pull/4915#issuecomment-1168769706
We'll have to wait for @rdblue or another committer to review it and merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org