You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jackylee-ch (via GitHub)" <gi...@apache.org> on 2026/03/28 09:07:40 UTC

[PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

jackylee-ch opened a new pull request, #15814:
URL: https://github.com/apache/iceberg/pull/15814

   When using a Hive catalog, schema evolution operations that change column positions — such as ALTER TABLE ... DROP COLUMN (non-last column) and ALTER TABLE ... ALTER COLUMN ... FIRST/AFTER (reorder) — fail with InvalidOperationException from MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange. This is because the Hive Metastore validates schema changes by comparing column types positionally (by index, not by name), controlled by hive.metastore.disallow.incompatible.col.type.changes (default true).
   
     This limitation is not documented anywhere in the Iceberg docs, though the Iceberg test suite itself works around it by setting METASTORE_DISALLOW_INCOMPATIBLE_COL_TYPE_CHANGES=false (TestHiveMetastore.java:269), and a test comment(HiveTableTest.java:283-287) explicitly describes the issue.
   
     This PR adds !!! warning admonition blocks to:
     - spark-ddl.md — under the DROP COLUMN section (full explanation, workaround, and trade-off) and the ALTER COLUMN reorder section (with cross-reference)
     - flink-ddl.md — under the Hive catalog configuration section (engine-agnostic warning)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3006432738


##########
docs/docs/spark-ddl.md:
##########
@@ -326,6 +330,21 @@ ALTER TABLE prod.db.sample DROP COLUMN id;
 ALTER TABLE prod.db.sample DROP COLUMN point.z;
 ```
 
+!!! warning "Hive Catalog Limitation"
+    When using a **Hive catalog**, dropping a column that is not the last column in the table schema
+    may fail with an error from the Hive Metastore (HMS). This occurs because HMS validates schema
+    changes by comparing column types **positionally** — dropping a middle column shifts subsequent
+    columns, which HMS interprets as incompatible type changes
+    (`MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange`).
+
+    To work around this, disable the HMS schema compatibility check by setting
+    `hive.metastore.disallow.incompatible.col.type.changes=false` in `hive-site.xml`, or by passing
+    `--conf spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false` when starting Spark.

Review Comment:
   can not recall the detail, but I remember that overriding this config on the client-side does not work on all HMS versions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3007810628


##########
docs/docs/spark-ddl.md:
##########
@@ -326,6 +330,21 @@ ALTER TABLE prod.db.sample DROP COLUMN id;
 ALTER TABLE prod.db.sample DROP COLUMN point.z;
 ```
 
+!!! warning "Hive Catalog Limitation"
+    When using a **Hive catalog**, dropping a column that is not the last column in the table schema
+    may fail with an error from the Hive Metastore (HMS). This occurs because HMS validates schema
+    changes by comparing column types **positionally** — dropping a middle column shifts subsequent
+    columns, which HMS interprets as incompatible type changes
+    (`MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange`).
+
+    To work around this, disable the HMS schema compatibility check by setting
+    `hive.metastore.disallow.incompatible.col.type.changes=false` in `hive-site.xml`, or by passing
+    `--conf spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false` when starting Spark.

Review Comment:
   back here, I get a response that config `hive.metastore.disallow.incompatible.col.type.changes` client-side overriding requires HIVE-17832, HIVE-17942



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3056019695


##########
docs/docs/flink-ddl.md:
##########
@@ -45,6 +45,23 @@ The following properties can be set if using the Hive catalog:
 * `hive-conf-dir`: Path to a directory containing a `hive-site.xml` configuration file which will be used to provide custom Hive configuration values. The value of `hive.metastore.warehouse.dir` from `<hive-conf-dir>/hive-site.xml` (or hive configure file from classpath) will be overwritten with the `warehouse` value if setting both `hive-conf-dir` and `warehouse` when creating iceberg catalog.
 * `hadoop-conf-dir`: Path to a directory containing `core-site.xml` and `hdfs-site.xml` configuration files which will be used to provide custom Hadoop configuration values.
 
+!!! warning "Hive Catalog Limitation"
+    The Hive Metastore validates schema changes by comparing column types **positionally**

Review Comment:
   ```suggestion
       The Hive Metastore (HMS) validates schema changes by comparing column types **positionally**
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3007075020


##########
docs/docs/spark-ddl.md:
##########
@@ -326,6 +330,21 @@ ALTER TABLE prod.db.sample DROP COLUMN id;
 ALTER TABLE prod.db.sample DROP COLUMN point.z;
 ```
 
+!!! warning "Hive Catalog Limitation"
+    When using a **Hive catalog**, dropping a column that is not the last column in the table schema
+    may fail with an error from the Hive Metastore (HMS). This occurs because HMS validates schema
+    changes by comparing column types **positionally** — dropping a middle column shifts subsequent
+    columns, which HMS interprets as incompatible type changes
+    (`MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange`).
+
+    To work around this, disable the HMS schema compatibility check by setting
+    `hive.metastore.disallow.incompatible.col.type.changes=false` in `hive-site.xml`, or by passing
+    `--conf spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false` when starting Spark.

Review Comment:
   Hm, I have tried it for hive 2.3.9 and 3.1.3. Do you remember the HMS version you tried that failed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#issuecomment-4212009368

   cc @manuzhang PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3016526895


##########
docs/docs/spark-ddl.md:
##########
@@ -326,6 +330,21 @@ ALTER TABLE prod.db.sample DROP COLUMN id;
 ALTER TABLE prod.db.sample DROP COLUMN point.z;
 ```
 
+!!! warning "Hive Catalog Limitation"
+    When using a **Hive catalog**, dropping a column that is not the last column in the table schema
+    may fail with an error from the Hive Metastore (HMS). This occurs because HMS validates schema
+    changes by comparing column types **positionally** — dropping a middle column shifts subsequent
+    columns, which HMS interprets as incompatible type changes
+    (`MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange`).
+
+    To work around this, disable the HMS schema compatibility check by setting
+    `hive.metastore.disallow.incompatible.col.type.changes=false` in `hive-site.xml`, or by passing
+    `--conf spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false` when starting Spark.

Review Comment:
   I have test with flink 1.20 and hive client 3.1.3, it doesn't work.
   
   The hms client also needs to actively use `setMetaConf` to make it effective. However, neither Spark, Flink, nor Iceberg currently offer this type of operation, so configuring it directly in the job will not work.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3036642661


##########
docs/docs/flink-ddl.md:
##########
@@ -45,6 +45,23 @@ The following properties can be set if using the Hive catalog:
 * `hive-conf-dir`: Path to a directory containing a `hive-site.xml` configuration file which will be used to provide custom Hive configuration values. The value of `hive.metastore.warehouse.dir` from `<hive-conf-dir>/hive-site.xml` (or hive configure file from classpath) will be overwritten with the `warehouse` value if setting both `hive-conf-dir` and `warehouse` when creating iceberg catalog.
 * `hadoop-conf-dir`: Path to a directory containing `core-site.xml` and `hdfs-site.xml` configuration files which will be used to provide custom Hadoop configuration values.
 
+!!! warning "Hive Catalog Limitation"
+    The Hive Metastore validates schema changes by comparing column types **positionally**
+    (`hive.metastore.disallow.incompatible.col.type.changes`, default `true`). When using a Hive catalog,
+    schema evolution operations that change column positions — such as dropping a non-last column or
+    reordering columns — may fail regardless of which engine performs the change (Spark, Flink Java API, etc.).
+
+    To work around this, disable the HMS schema compatibility check by setting
+    `hive.metastore.disallow.incompatible.col.type.changes=false`:
+
+    - **Remote HMS:** Set this property in the HMS server's `hive-site.xml`.
+    - **Embedded HMS:** Add the equivalent property to the Hive catalog configuration.

Review Comment:
   The `Embedded HMS` means using HMS with derby, which is launching in the Flink JM.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3014373053


##########
docs/docs/spark-ddl.md:
##########
@@ -326,6 +330,21 @@ ALTER TABLE prod.db.sample DROP COLUMN id;
 ALTER TABLE prod.db.sample DROP COLUMN point.z;
 ```
 
+!!! warning "Hive Catalog Limitation"
+    When using a **Hive catalog**, dropping a column that is not the last column in the table schema
+    may fail with an error from the Hive Metastore (HMS). This occurs because HMS validates schema
+    changes by comparing column types **positionally** — dropping a middle column shifts subsequent
+    columns, which HMS interprets as incompatible type changes
+    (`MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange`).
+
+    To work around this, disable the HMS schema compatibility check by setting
+    `hive.metastore.disallow.incompatible.col.type.changes=false` in `hive-site.xml`, or by passing
+    `--conf spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false` when starting Spark.

Review Comment:
   I have check remote HMS with version 3.1.3, which contains the HIVE-17832 and HIVE-17942, they still don't work. Thus, I updated the documentation as follows.
       - **Remote HMS:** Set this property in the HMS server's `hive-site.xml`.
       - **Embedded HMS:** Add the equivalent property to the Hive catalog configuration.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3014384601


##########
docs/docs/spark-ddl.md:
##########
@@ -326,6 +330,21 @@ ALTER TABLE prod.db.sample DROP COLUMN id;
 ALTER TABLE prod.db.sample DROP COLUMN point.z;
 ```
 
+!!! warning "Hive Catalog Limitation"
+    When using a **Hive catalog**, dropping a column that is not the last column in the table schema
+    may fail with an error from the Hive Metastore (HMS). This occurs because HMS validates schema
+    changes by comparing column types **positionally** — dropping a middle column shifts subsequent
+    columns, which HMS interprets as incompatible type changes
+    (`MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange`).
+
+    To work around this, disable the HMS schema compatibility check by setting
+    `hive.metastore.disallow.incompatible.col.type.changes=false` in `hive-site.xml`, or by passing
+    `--conf spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false` when starting Spark.

Review Comment:
   you should apply these patches to the HMS client ...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3006432738


##########
docs/docs/spark-ddl.md:
##########
@@ -326,6 +330,21 @@ ALTER TABLE prod.db.sample DROP COLUMN id;
 ALTER TABLE prod.db.sample DROP COLUMN point.z;
 ```
 
+!!! warning "Hive Catalog Limitation"
+    When using a **Hive catalog**, dropping a column that is not the last column in the table schema
+    may fail with an error from the Hive Metastore (HMS). This occurs because HMS validates schema
+    changes by comparing column types **positionally** — dropping a middle column shifts subsequent
+    columns, which HMS interprets as incompatible type changes
+    (`MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange`).
+
+    To work around this, disable the HMS schema compatibility check by setting
+    `hive.metastore.disallow.incompatible.col.type.changes=false` in `hive-site.xml`, or by passing
+    `--conf spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false` when starting Spark.

Review Comment:
   can not recall the detail, but I remember that overriding this config on the client-side does not work on some HMS versions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3007105637


##########
docs/docs/spark-ddl.md:
##########
@@ -326,6 +330,21 @@ ALTER TABLE prod.db.sample DROP COLUMN id;
 ALTER TABLE prod.db.sample DROP COLUMN point.z;
 ```
 
+!!! warning "Hive Catalog Limitation"
+    When using a **Hive catalog**, dropping a column that is not the last column in the table schema
+    may fail with an error from the Hive Metastore (HMS). This occurs because HMS validates schema
+    changes by comparing column types **positionally** — dropping a middle column shifts subsequent
+    columns, which HMS interprets as incompatible type changes
+    (`MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange`).
+
+    To work around this, disable the HMS schema compatibility check by setting
+    `hive.metastore.disallow.incompatible.col.type.changes=false` in `hive-site.xml`, or by passing
+    `--conf spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false` when starting Spark.

Review Comment:
   did you test with embedded HMS, or remote HMS? IIRC, Hive 2.3.9 with remote HMS does not work. I will contact our Hive team and sync the result today.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#issuecomment-4226663564

   Thanks @jackylee-ch for the PR! Thanks @pan3793 @wypoon @manuzhang for the review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#issuecomment-4184257028

   LGTM. I will leave the PR open for a couple of more days in case @manuzhang or other want to take a look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#issuecomment-4212157104

   LGTM with minor suggestions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3019444313


##########
docs/docs/spark-ddl.md:
##########
@@ -326,6 +330,21 @@ ALTER TABLE prod.db.sample DROP COLUMN id;
 ALTER TABLE prod.db.sample DROP COLUMN point.z;
 ```
 
+!!! warning "Hive Catalog Limitation"
+    When using a **Hive catalog**, dropping a column that is not the last column in the table schema
+    may fail with an error from the Hive Metastore (HMS). This occurs because HMS validates schema
+    changes by comparing column types **positionally** — dropping a middle column shifts subsequent
+    columns, which HMS interprets as incompatible type changes
+    (`MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange`).
+
+    To work around this, disable the HMS schema compatibility check by setting
+    `hive.metastore.disallow.incompatible.col.type.changes=false` in `hive-site.xml`, or by passing
+    `--conf spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false` when starting Spark.

Review Comment:
   @jackylee-ch, I see, and have double-checked with our hive team, there are additional internal changes to make client-side overriding work. Sorry about the incomplete information.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3035749947


##########
docs/docs/spark-ddl.md:
##########
@@ -259,6 +280,11 @@ ALTER TABLE prod.db.sample
 ADD COLUMN nested.new_column bigint FIRST;
 ```
 
+!!! warning "Hive Catalog Limitation"
+    When using a Hive catalog, adding a column with `FIRST` or `AFTER` may fail due to HMS positional
+    schema validation. See the [Hive Catalog Limitation](#hive-catalog-limitation) above for details

Review Comment:
   The link target `#hive-catalog-limitation` likely won’t resolve because the earlier “Hive Catalog Limitation” text is an admonition, not a heading/anchor that most MkDocs setups include in the page TOC. Add an explicit anchor (e.g., a `### Hive catalog limitation {#hive-catalog-limitation}` heading or an HTML anchor) so this cross-reference works reliably.
   ```suggestion
       schema validation. See the warning above for details
   ```



##########
docs/docs/spark-ddl.md:
##########
@@ -302,6 +328,10 @@ ALTER TABLE prod.db.sample ALTER COLUMN col FIRST;
 ALTER TABLE prod.db.sample ALTER COLUMN nested.col AFTER other_col;
 ```
 
+!!! warning "Hive Catalog Limitation"
+    When using a Hive catalog, reordering columns may fail due to HMS positional schema validation.
+    See the [Hive Catalog Limitation](#hive-catalog-limitation) above for details and workaround.

Review Comment:
   This cross-reference uses `#hive-catalog-limitation`, but the page doesn’t currently define that fragment as a heading/anchor (the earlier block is an admonition). Add an explicit anchor at the definition site or adjust the link to point to an existing heading so the link isn’t broken.
   ```suggestion
       See the Hive Catalog Limitation note above for details and workaround.
   ```



##########
docs/docs/spark-ddl.md:
##########
@@ -323,6 +353,11 @@ ALTER TABLE prod.db.sample DROP COLUMN id;
 ALTER TABLE prod.db.sample DROP COLUMN point.z;
 ```
 
+!!! warning "Hive Catalog Limitation"
+    When using a Hive catalog, dropping a non-last column may fail due to HMS positional schema
+    validation. See the [Hive Catalog Limitation](#hive-catalog-limitation) above for details
+    and workaround.

Review Comment:
   This warning links to `#hive-catalog-limitation`, but the document doesn’t define that fragment as a heading/anchor (the earlier “Hive Catalog Limitation” block is an admonition). Please add an explicit anchor or update the link to an existing section header to avoid a broken link.
   ```suggestion
       validation. See the earlier Hive Catalog Limitation warning above for details and
       workaround.
   ```



##########
docs/docs/flink-ddl.md:
##########
@@ -45,6 +45,23 @@ The following properties can be set if using the Hive catalog:
 * `hive-conf-dir`: Path to a directory containing a `hive-site.xml` configuration file which will be used to provide custom Hive configuration values. The value of `hive.metastore.warehouse.dir` from `<hive-conf-dir>/hive-site.xml` (or hive configure file from classpath) will be overwritten with the `warehouse` value if setting both `hive-conf-dir` and `warehouse` when creating iceberg catalog.
 * `hadoop-conf-dir`: Path to a directory containing `core-site.xml` and `hdfs-site.xml` configuration files which will be used to provide custom Hadoop configuration values.
 
+!!! warning "Hive Catalog Limitation"
+    The Hive Metastore validates schema changes by comparing column types **positionally**
+    (`hive.metastore.disallow.incompatible.col.type.changes`, default `true`). When using a Hive catalog,
+    schema evolution operations that change column positions — such as dropping a non-last column or
+    reordering columns — may fail regardless of which engine performs the change (Spark, Flink Java API, etc.).
+
+    To work around this, disable the HMS schema compatibility check by setting

Review Comment:
   The warning text uses the abbreviation “HMS” later, but it isn’t introduced/defined here. Either introduce it as “Hive Metastore (HMS)” on first use or avoid the acronym to keep the docs self-contained.
   ```suggestion
       To work around this, disable the Hive Metastore (HMS) schema compatibility check by setting
   ```



##########
docs/docs/flink-ddl.md:
##########
@@ -45,6 +45,23 @@ The following properties can be set if using the Hive catalog:
 * `hive-conf-dir`: Path to a directory containing a `hive-site.xml` configuration file which will be used to provide custom Hive configuration values. The value of `hive.metastore.warehouse.dir` from `<hive-conf-dir>/hive-site.xml` (or hive configure file from classpath) will be overwritten with the `warehouse` value if setting both `hive-conf-dir` and `warehouse` when creating iceberg catalog.
 * `hadoop-conf-dir`: Path to a directory containing `core-site.xml` and `hdfs-site.xml` configuration files which will be used to provide custom Hadoop configuration values.
 
+!!! warning "Hive Catalog Limitation"
+    The Hive Metastore validates schema changes by comparing column types **positionally**
+    (`hive.metastore.disallow.incompatible.col.type.changes`, default `true`). When using a Hive catalog,
+    schema evolution operations that change column positions — such as dropping a non-last column or
+    reordering columns — may fail regardless of which engine performs the change (Spark, Flink Java API, etc.).
+
+    To work around this, disable the HMS schema compatibility check by setting
+    `hive.metastore.disallow.incompatible.col.type.changes=false`:
+
+    - **Remote HMS:** Set this property in the HMS server's `hive-site.xml`.
+    - **Embedded HMS:** Add the equivalent property to the Hive catalog configuration.

Review Comment:
   The “Embedded HMS” workaround is unclear in the context of this Flink Hive catalog section (where `uri` is required and configuration is typically provided via `hive-conf-dir`/classpath). Consider rephrasing this to explicitly instruct users to set the property in a `hive-site.xml` picked up via `hive-conf-dir` (or classpath) rather than suggesting an “embedded” metastore configuration path that may not apply.
   ```suggestion
       - **When configuring the Hive catalog in Flink:** Set this property in a `hive-site.xml`
         that Flink picks up via `hive-conf-dir` or from the classpath.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3056024752


##########
docs/docs/flink-ddl.md:
##########
@@ -45,6 +45,23 @@ The following properties can be set if using the Hive catalog:
 * `hive-conf-dir`: Path to a directory containing a `hive-site.xml` configuration file which will be used to provide custom Hive configuration values. The value of `hive.metastore.warehouse.dir` from `<hive-conf-dir>/hive-site.xml` (or hive configure file from classpath) will be overwritten with the `warehouse` value if setting both `hive-conf-dir` and `warehouse` when creating iceberg catalog.
 * `hadoop-conf-dir`: Path to a directory containing `core-site.xml` and `hdfs-site.xml` configuration files which will be used to provide custom Hadoop configuration values.
 
+!!! warning "Hive Catalog Limitation"
+    The Hive Metastore validates schema changes by comparing column types **positionally**
+    (`hive.metastore.disallow.incompatible.col.type.changes`, default `true`). When using a Hive catalog,
+    schema evolution operations that change column positions — such as dropping a non-last column or
+    reordering columns — may fail regardless of which engine performs the change (Spark, Flink Java API, etc.).
+
+    To work around this, disable the Hive Metastore (HMS) schema compatibility check by setting

Review Comment:
   ```suggestion
       To work around this, disable the HMS schema compatibility check by setting
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao merged PR #15814:
URL: https://github.com/apache/iceberg/pull/15814


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#issuecomment-4182208095

   cc @manuzhang @huaxingao 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3014497213


##########
docs/docs/spark-ddl.md:
##########
@@ -326,6 +330,21 @@ ALTER TABLE prod.db.sample DROP COLUMN id;
 ALTER TABLE prod.db.sample DROP COLUMN point.z;
 ```
 
+!!! warning "Hive Catalog Limitation"
+    When using a **Hive catalog**, dropping a column that is not the last column in the table schema
+    may fail with an error from the Hive Metastore (HMS). This occurs because HMS validates schema
+    changes by comparing column types **positionally** — dropping a middle column shifts subsequent
+    columns, which HMS interprets as incompatible type changes
+    (`MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange`).
+
+    To work around this, disable the HMS schema compatibility check by setting
+    `hive.metastore.disallow.incompatible.col.type.changes=false` in `hive-site.xml`, or by passing
+    `--conf spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false` when starting Spark.

Review Comment:
   > you should apply these patches to the HMS client ...
   
   Oh. got it, would make a double check for hive client 3.x



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3019420746


##########
docs/docs/spark-ddl.md:
##########


Review Comment:
   Good catch. I missed that we can reorder columns with ADD COLUMN and ALTER COLUMN, I'll reconsider how this part of the documentation is written.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "wypoon (via GitHub)" <gi...@apache.org>.
wypoon commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3017844374


##########
docs/docs/spark-ddl.md:
##########


Review Comment:
   This needs a similar warning too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Docs: Add Hive Metastore schema validation warnings for schema evolution with Hive catalog [iceberg]

Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #15814:
URL: https://github.com/apache/iceberg/pull/15814#discussion_r3007133429


##########
docs/docs/spark-ddl.md:
##########
@@ -326,6 +330,21 @@ ALTER TABLE prod.db.sample DROP COLUMN id;
 ALTER TABLE prod.db.sample DROP COLUMN point.z;
 ```
 
+!!! warning "Hive Catalog Limitation"
+    When using a **Hive catalog**, dropping a column that is not the last column in the table schema
+    may fail with an error from the Hive Metastore (HMS). This occurs because HMS validates schema
+    changes by comparing column types **positionally** — dropping a middle column shifts subsequent
+    columns, which HMS interprets as incompatible type changes
+    (`MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange`).
+
+    To work around this, disable the HMS schema compatibility check by setting
+    `hive.metastore.disallow.incompatible.col.type.changes=false` in `hive-site.xml`, or by passing
+    `--conf spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false` when starting Spark.

Review Comment:
   I was testing with embedded HMS. I just asks the colleague in charge of Hive, these should be configured in the metadata store, unless user use embedded HMS just like I did.
   
   I would update the doc, thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org