[GitHub] [iceberg] wypoon edited a comment on pull request #4017: Write engine.hive.enabled=true in table properties for Hive-enabled table

[GitBox <gi...@apache.org>]

wypoon edited a comment on pull request #4017:
URL: https://github.com/apache/iceberg/pull/4017#issuecomment-1028233193


   @rdblue it is true that if your Hadoop `Configuration` has iceberg.engine.hive.enabled set to true for all your writers, this is not an issue. I'd argue that as a matter of logical consistency, the table property should be set if the table is created as Hive-enabled, so that a future writer (whose configuration we may not be in control of) that updates the table does not drop that Hive-enabledness.
   
   [In `HiveTableOperations#doCommit`, we check if `hiveEngineEnabled` by checking if "engine.hive.enabled" is in the `TableMetadata`'s properties and using its value if so, else use the value of "iceberg.engine.hive.enabled" in the Hadoop `Configuration`, **defaulting to false**. Thus there are two routes to `hiveEngineEnabled` being true, because "engine.hive.enabled" is in the table properties, or because of the Hadoop `Configuration`. What I'm proposing is that we always then encode it in the table properties, so future commits will keep the table being Hive-enabled, regardless of the writer's Hadoop `Configuration`. Otherwise, `hiveEngineEnabled` could be true due to the `Configuration`, and not write the property to `TableMetadata`, and another writer comes along with a different `Configuration`, and `hiveEngineEnabled` is now false from the `TableMetadata`.]
   
   We encountered the issue in a setting with a shared HMS serving multiple clusters.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org