You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/11/11 17:25:16 UTC
[GitHub] [iceberg] boroknagyz opened a new pull request #1751: Set Input/Output formats for the HMS storage descriptor
boroknagyz opened a new pull request #1751:
URL: https://github.com/apache/iceberg/pull/1751
Some engines (e.g. Impala) depend on the Input/Output format classes
being set in the HMS storage descriptor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1751: Set Input/Output formats for the HMS storage descriptor
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1751:
URL: https://github.com/apache/iceberg/pull/1751#issuecomment-725593828
@boroknagyz, the problem with setting this is that Hive doesn't include these classes by default and missing classes will break Hive. We've added configuration to set up the Hive storage descriptor properly to both the Hive configuration (`iceberg.engine.hive.enabled`) and to table properties (`engine.hive.enabled`). If you enable Hive either way, then the `IcebergHiveStorageHandler` is set up.
I also would not recommend any engine use the Hive configuration other than Hive. I think Impala is planning to add native read support for Iceberg tables, which would be a much better way to read.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] boroknagyz commented on pull request #1751: Set Input/Output formats for the HMS storage descriptor
Posted by GitBox <gi...@apache.org>.
boroknagyz commented on pull request #1751:
URL: https://github.com/apache/iceberg/pull/1751#issuecomment-725688314
Thanks, @rdblue !
Btw when I said Impala "already supports this and that", I meant it's on the master branch, but we haven't released anything yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #1751: Set Input/Output formats for the HMS storage descriptor
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #1751:
URL: https://github.com/apache/iceberg/pull/1751
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1751: Set Input/Output formats for the HMS storage descriptor
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1751:
URL: https://github.com/apache/iceberg/pull/1751#issuecomment-725679188
@boroknagyz, sorry for the confusion. I thought that this was setting the formats for the path where the Hive engine is not enabled. Since this is setting only when Hive is enabled, I think it is safe. Thanks for pointing out the SerDe, that's what made me realize the error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] boroknagyz commented on pull request #1751: Set Input/Output formats for the HMS storage descriptor
Posted by GitBox <gi...@apache.org>.
boroknagyz commented on pull request #1751:
URL: https://github.com/apache/iceberg/pull/1751#issuecomment-725676838
Thanks @rdblue for your response. Impala already has read support for Iceberg tables (that are stored in HadoopTables or HadoopCatalog). (Btw Impala can already write non-partitioned Iceberg tables).
Impala uses HMS to store table metadata and uses the InputFormat class of the storage descriptor to determine the table format. So it doesn't use these classes, but only their names to identify the table type.
Currently Impala uses class names from Hiveberg to identify Iceberg tables:
https://github.com/apache/impala/blob/ea13e744977c71206c11a54f65d77aeecb55703d/fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java#L80
But there's already an upstream change request to use the new classes from Iceberg:
https://gerrit.cloudera.org/#/c/16707/
I don't think this change would introduce missing classes in Hive because HiveTableOperations already sets org.apache.iceberg.mr.hive.HiveIcebergSerDe which is in the same package.
I think this change would help interoperability between Impala and Hive as they would create tables the same way, What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #1751: Set Input/Output formats for the HMS storage descriptor
Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #1751:
URL: https://github.com/apache/iceberg/pull/1751#issuecomment-725871678
Thanks @rdblue for the notification! Discussed this with @boroknagyz offline, and this change is fine for the Hive too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1751: Set Input/Output formats for the HMS storage descriptor
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1751:
URL: https://github.com/apache/iceberg/pull/1751#issuecomment-725679381
@pvary FYI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org