You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/11/10 12:31:24 UTC

[GitHub] [iceberg] massdosage opened a new pull request #1748: Hive read via HiveCatalog documentation

massdosage opened a new pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] rdblue commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r521637341



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,19 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive. 
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`.
+TODO: what do we need to set up when we create this table programatically for everything to be registered correctly for read usage in Hive?

Review comment:
       The logic for setting up the storage handler is here: https://github.com/apache/iceberg/blob/master/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L377-L384
   
   It uses both the environment config and the table property.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] rdblue merged pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] rdsr commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r522302920



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,38 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive.
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`. In order for Iceberg to correctly set up the Hive table for querying some configuration values need to be set, the two options for this are described below - you can use either or the other depending on your use case.
+
+##### Hive Configuration
+The value `iceberg.engine.hive.enabled` needs to be set to `true` and added to the Hive configuration file on the classpath of the application creating the table. This can be done by modifying the relevant `hive-site.xml`. Alternatively this can done programatically like so:

Review comment:
       I assume we can query existing Iceberg tables also through Hive by setting the `iceberg.engine.hive.enabled` flag? Do you think it worth calling that out?

##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,38 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive.
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`. In order for Iceberg to correctly set up the Hive table for querying some configuration values need to be set, the two options for this are described below - you can use either or the other depending on your use case.
+
+##### Hive Configuration
+The value `iceberg.engine.hive.enabled` needs to be set to `true` and added to the Hive configuration file on the classpath of the application creating the table. This can be done by modifying the relevant `hive-site.xml`. Alternatively this can done programatically like so:
+```java
+Configuration hadoopConfiguration = spark.sparkContext().hadoopConfiguration();
+hadoopConfiguration.set(ConfigProperties.ENGINE_HIVE_ENABLED, "true"); //iceberg.engine.hive.enabled=true
+HiveCatalog catalog = new HiveCatalog(hadoopConfiguration);
+...
+catalog.createTable(tableId, schema, spec);
+```
+
+##### Table Property Configuration
+The property `engine.hive.enabled` needs to be set to `true` and added to the table properties when creating the Iceberg table. This can be done like so:
+```java
+    Map<String, String> tableProperties = new HashMap<String, String>();
+    tableProperties.put(TableProperties.ENGINE_HIVE_ENABLED, "true"); ////engine.hive.enabled=true
+    catalog.createTable(tableId, schema, spec, tableProperties);
+```
+
+#### Query the Iceberg table via Hive
+TODO: tables created by the above can't just be read "as is", need to document steps needed in order to be able to query them here.

Review comment:
       This seems to me contradictory to the following lines. Is this TODO still valid?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] rdblue commented on pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#issuecomment-730755423


   Thanks @massdosage! Great to have this documented.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] massdosage commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r520692681



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,19 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive. 
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`.
+TODO: what do we need to set up when we create this table programatically for everything to be registered correctly for read usage in Hive?

Review comment:
       OK, so, with setting that property the DDL looks better:
   ```
   CREATE EXTERNAL TABLE `iceberg_table_from_hive_catalog`(
   )
   ROW FORMAT SERDE 
     'org.apache.iceberg.mr.hive.HiveIcebergSerDe' 
   STORED BY 
     'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler' 
   
   LOCATION
     's3://REDACTED/iceberg_table_from_hive_catalog'
   TBLPROPERTIES (
     'metadata_location'='s3://REDACTED/iceberg_table_from_hive_catalog/metadata/00000-95297465-a383-4f0b-b546-dffe7c4ff778.metadata.json', 
     'table_type'='ICEBERG', 
     'transient_lastDdlTime'='1605024789')
   ```
   Not sure whether it's an issue that the schema information is missing? I also can't successfully query this table from Hive so I'll look into that next and see what the issue is. Thanks for helping me over the first hurdle though, I'll update this PR shortly with that required config setting.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] pvary commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r520567223



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,19 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive. 
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`.
+TODO: what do we need to set up when we create this table programatically for everything to be registered correctly for read usage in Hive?

Review comment:
       I think setting `engine.hive.enabled=true` should help.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] massdosage commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r522354227



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,38 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive.
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`. In order for Iceberg to correctly set up the Hive table for querying some configuration values need to be set, the two options for this are described below - you can use either or the other depending on your use case.
+
+##### Hive Configuration
+The value `iceberg.engine.hive.enabled` needs to be set to `true` and added to the Hive configuration file on the classpath of the application creating the table. This can be done by modifying the relevant `hive-site.xml`. Alternatively this can done programatically like so:

Review comment:
       I'll let you know if and when I get it working ;) and yes, that will need to be added to the documentation.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] massdosage commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r520660905



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,19 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive. 
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`.
+TODO: what do we need to set up when we create this table programatically for everything to be registered correctly for read usage in Hive?

Review comment:
       OK, so I tried adding this:
   ```
   hadoopConfiguration.set("engine.hive.enabled", "true");
   ```
   before passing the `hadoopConfiguration` on to the
   ```
   HiveCatalog catalog = new HiveCatalog(hadoopConfiguration);
   ```
   but it doesn't seem to make any difference to the DDL of the table that gets subsequently created.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] massdosage commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r520682236



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,19 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive. 
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`.
+TODO: what do we need to set up when we create this table programatically for everything to be registered correctly for read usage in Hive?

Review comment:
       Oh wait, `iceberg.engine.hive.enabled` is what it should be, let me try that...




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] massdosage commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r521922877



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,19 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive. 
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`.
+TODO: what do we need to set up when we create this table programatically for everything to be registered correctly for read usage in Hive?

Review comment:
       Yes, exactly, so in this case when I'm creating a new table it doesn't find the value in the table metadata so it drops through to line 383 and looks in the `Configuration` using the `ConfigProperties` key I mentioned above which is `iceberg.engine.hive.enabled`. I stepped through this code to figure out why it wasn't working with `engine.hive.enabled` and saw the property it was looking for in the conf is actually `iceberg.engine.hive.enabled`. So I guess there are two options here:
   
   1. you explicitly create the table with metadata and set `engine.hive.enabled` as a table property.
   
   or 
   
   2. you set the value `iceberg.engine.hive.enabled` explicitly in the conf object (like I'm doing in my example code above) or let this be automatically set by adding it to `hive-site.xml`.
   
   If we agree on this I can document the above here ^. I
   
   I'll then look into what's necessary in order to query the table and add it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] massdosage commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r522353883



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,38 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive.
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`. In order for Iceberg to correctly set up the Hive table for querying some configuration values need to be set, the two options for this are described below - you can use either or the other depending on your use case.
+
+##### Hive Configuration
+The value `iceberg.engine.hive.enabled` needs to be set to `true` and added to the Hive configuration file on the classpath of the application creating the table. This can be done by modifying the relevant `hive-site.xml`. Alternatively this can done programatically like so:
+```java
+Configuration hadoopConfiguration = spark.sparkContext().hadoopConfiguration();
+hadoopConfiguration.set(ConfigProperties.ENGINE_HIVE_ENABLED, "true"); //iceberg.engine.hive.enabled=true
+HiveCatalog catalog = new HiveCatalog(hadoopConfiguration);
+...
+catalog.createTable(tableId, schema, spec);
+```
+
+##### Table Property Configuration
+The property `engine.hive.enabled` needs to be set to `true` and added to the table properties when creating the Iceberg table. This can be done like so:
+```java
+    Map<String, String> tableProperties = new HashMap<String, String>();
+    tableProperties.put(TableProperties.ENGINE_HIVE_ENABLED, "true"); ////engine.hive.enabled=true
+    catalog.createTable(tableId, schema, spec, tableProperties);
+```
+
+#### Query the Iceberg table via Hive
+TODO: tables created by the above can't just be read "as is", need to document steps needed in order to be able to query them here.

Review comment:
       Yes, I'm currently unable to query the tables created by the above from Hive and need to figure out what I need to do in order to get it working. There's either an issue with the generated DDL or some properties need to be set at read time. I haven't had time to look into it properly yet.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] massdosage commented on pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
massdosage commented on pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#issuecomment-731046455


   > Thanks @massdosage! Great to have this documented.
   
   Sure, I'll hopefully have time in the next week or so to document the HadoopCataog and CustomCatalog paths.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] massdosage commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r520528074



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,19 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive. 
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`.
+TODO: what do we need to set up when we create this table programatically for everything to be registered correctly for read usage in Hive?

Review comment:
       @pvary What does one need to do in order to get the table set up properly for the Hive read path in this case? What I have tried to do so far is this, first create an Iceberg table using the HiveCatalog like so:
   ```
    PartitionSpec spec = PartitionSpec.unpartitioned();
    Schema schema = new Schema(optional(1, "id", Types.LongType.get()), optional(2, "name", Types.StringType.get()));
    SparkSession spark = SparkSession.builder().appName("IcebergTest").getOrCreate();
    Configuration hadoopConfiguration = spark.sparkContext().hadoopConfiguration();
    Catalog catalog = new HiveCatalog(hadoopConfiguration);
    TableIdentifier tableId = TableIdentifier.of("test", "iceberg_table_from_hive_catalog");
    catalog.createTable(tableId, schema, spec);
   ```
   The table created in Hive by the above has DDL like so:
   ```
   CREATE EXTERNAL TABLE `iceberg_table_from_hive_catalog`(
     `id` bigint COMMENT '', 
     `name` string COMMENT '')
   ROW FORMAT SERDE 
     'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
   STORED AS INPUTFORMAT 
     'org.apache.hadoop.mapred.FileInputFormat' 
   OUTPUTFORMAT 
     'org.apache.hadoop.mapred.FileOutputFormat'
   LOCATION
     's3://REDACTED/iceberg_table_from_hive_catalog'
   TBLPROPERTIES (
     'metadata_location'='s3://REDACTED/iceberg_table_from_hive_catalog/metadata/00000-7addbbf2-1836-4973-86af-0511ae7577fb.metadata.json', 
     'table_type'='ICEBERG', 
     'transient_lastDdlTime'='1605007216')
   ```
   Which is obviously incorrect as the StorageHandler hasn't been set etc.. I know you worked on a PR that set this all up properly as long as some config/setup was performed at table creation time. Can you please let me know what I need to do and I'll then document it accordingly once I test it working?
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] pvary commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r521926244



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,19 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive. 
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`.
+TODO: what do we need to set up when we create this table programatically for everything to be registered correctly for read usage in Hive?

Review comment:
       CC @lcspinter as he is working on coming up a generic way to handle properties / configuration values wrt Hive.
   
   We would like to come up with solution where the user could easily understand how the different properties are used:
   - Iceberg table properties
   - HMS table properties
   - HMS serde properties
   - HiveConf
   - Default values
   
   Also we should understand how these properties are used when:
   - Reading Hive / Iceberg tables
   - Writing Hive / Iceberg tables




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] massdosage commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r520528074



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,19 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive. 
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`.
+TODO: what do we need to set up when we create this table programatically for everything to be registered correctly for read usage in Hive?

Review comment:
       @pvary What does one need to do in order to get the table set up properly for the Hive read path in this case? What I have tried to do so far is this, first create an Iceberg table using the HiveCatalog like so:
   ```
    PartitionSpec spec = PartitionSpec.unpartitioned();
    Schema schema = new Schema(optional(1, "id", Types.LongType.get()), optional(2, "name", Types.StringType.get()));
    SparkSession spark = SparkSession.builder().appName("IcebergTest").getOrCreate();
    Catalog catalog = new HiveCatalog(hadoopConfiguration);
    TableIdentifier tableId = TableIdentifier.of("test", "iceberg_table_from_hive_catalog");
    catalog.createTable(tableId, schema, spec);
   ```
   The table created in Hive by the above has DDL like so:
   ```
   CREATE EXTERNAL TABLE `iceberg_table_from_hive_catalog`(
     `id` bigint COMMENT '', 
     `name` string COMMENT '')
   ROW FORMAT SERDE 
     'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
   STORED AS INPUTFORMAT 
     'org.apache.hadoop.mapred.FileInputFormat' 
   OUTPUTFORMAT 
     'org.apache.hadoop.mapred.FileOutputFormat'
   LOCATION
     's3://REDACTED/iceberg_table_from_hive_catalog'
   TBLPROPERTIES (
     'metadata_location'='s3://REDACTED/iceberg_table_from_hive_catalog/metadata/00000-7addbbf2-1836-4973-86af-0511ae7577fb.metadata.json', 
     'table_type'='ICEBERG', 
     'transient_lastDdlTime'='1605007216')
   ```
   Which is obviously incorrect as the StorageHandler hasn't been set etc.. I know you worked on a PR that set this all up properly as long as some config/setup was performed at table creation time. Can you please let me know what I need to do and I'll then document it accordingly once I test it working?
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] massdosage commented on pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
massdosage commented on pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#issuecomment-727955138


   @pvary thanks for the above. For now my focus is getting a new HiveCatalog table created in such a way that it can be queried. I'll try some variation of what you suggest above with various Hive configuration settings and table properties and see if I can get it working - if so I'll update the steps in this document so you and others can confirm if that's correct, if not I'll add the exceptions I'm seeing here and we can discuss.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] massdosage commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r521570138



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,19 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive. 
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`.
+TODO: what do we need to set up when we create this table programatically for everything to be registered correctly for read usage in Hive?

Review comment:
       > Also, `engine.hive.enabled` is the table property. When a table has that property, it will be created with the storage handler. I set the property on my table from Spark and it worked.
   
   Really? From what I can see the code uses the value from here https://github.com/apache/iceberg/blob/b4030093b222a96a979d2964eecd6cdd3e9f7608/core/src/main/java/org/apache/iceberg/hadoop/ConfigProperties.java#L27 which is prefixed with `iceberg` as I said above. I tried it both ways and only this way worked but the schema in the Hive table definition is empty so Hive seems to be throwing an error about that when it tries to query it. I haven't tried setting `iceberg.mr.catalog=hive` in Hive when I do the query, I will try that next and see if it makes any difference and what impact that has on the HadoopTables version of it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] massdosage commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r526809400



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,38 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive.
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`. In order for Iceberg to correctly set up the Hive table for querying some configuration values need to be set, the two options for this are described below - you can use either or the other depending on your use case.
+
+##### Hive Configuration
+The value `iceberg.engine.hive.enabled` needs to be set to `true` and added to the Hive configuration file on the classpath of the application creating the table. This can be done by modifying the relevant `hive-site.xml`. Alternatively this can done programatically like so:

Review comment:
       Updated docs to describe the property you need to set for Hive read is `set iceberg.mr.catalog=hive`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] rdblue commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r521566746



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,19 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive. 
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`.
+TODO: what do we need to set up when we create this table programatically for everything to be registered correctly for read usage in Hive?

Review comment:
       I got this working when I tested 0.10.0-rc4. In Hive, I had to set `iceberg.mr.catalog=hive` so that the correct catalog was used. Unfortunately, that broke the Hadoop table.
   
   I think we might want to add support for Hadoop tables in our catalogs using an identifier that contains a path to fix this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] pvary commented on pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#issuecomment-726724055


   @massdosage: I try to answer all of the open questions - might miss some, so please feel free to point them out.
   
   - The property `iceberg.engine.hive.enabled`/`engine.hive.enabled` only controls how the HMS metadata values are set (InputFormat/OutputFormat/SerDe/StorageHandler)
   - There are different DDLs for different catalogs, these should work since they are tested with the unit tests as well
   ```
   -- empty iceberg.mr.catalog means HadoopTables
   
   -- new table Hive + Iceberg
   CREATE EXTERNAL TABLE customers 
   STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler'
   LOCATION '<LOCATION>'
   TBLPROPERTIES ('iceberg.mr.table.schema'='<SCHEMA_JSON>', 'iceberg.mr.table.partition.spec'='<PART_SPEC_JSON>');
   
   -- new Hive table above existing Iceberg table
   CREATE EXTERNAL TABLE customers 
   STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler'
   LOCATION '<LOCATION>';
   
   -- HadoopCatalog (same as HadoopTables)
   set iceberg.mr.catalog=hadoop;
   
   -- new table Hive + Iceberg
   CREATE EXTERNAL TABLE customers 
   STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler'
   LOCATION '<LOCATION>'
   TBLPROPERTIES ('iceberg.mr.table.schema'='<SCHEMA_JSON>', 'iceberg.mr.table.partition.spec'='<PART_SPEC_JSON>');
   
   -- new Hive table above existing Iceberg table
   CREATE EXTERNAL TABLE customers 
   STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler'
   LOCATION '<LOCATION>';
   
   -- HiveCatalog
   set iceberg.mr.catalog=hive;
   
   -- new table Hive + Iceberg
   CREATE EXTERNAL TABLE customers 
   STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler'
   TBLPROPERTIES ('iceberg.mr.table.schema'='<SCHEMA_JSON>', 'iceberg.mr.table.partition.spec'='<PART_SPEC_JSON>');
   
   -- new Hive table above existing Iceberg table is an invalid case
   ```
   - You might provide the Schema json when creating the table, but the property should not be present on the created HMS table.
   - You said that you have problem selecting data from the tables. What is the exact Exception?
   
   Thanks, and sorry for chiming in late 😢 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] rdblue commented on a change in pull request #1748: Hive read via HiveCatalog documentation

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1748:
URL: https://github.com/apache/iceberg/pull/1748#discussion_r521567251



##########
File path: site/docs/hive.md
##########
@@ -50,6 +50,19 @@ You should now be able to issue Hive SQL `SELECT` queries using the above table
 SELECT * from table_a;
 ```
 
+#### Using Hive Catalog
+Iceberg tables created using `HiveCatalog` are automatically registered with Hive. 
+
+##### Create an Iceberg table
+The first step is to create an Iceberg table using the Spark/Java/Python API and `HiveCatalog`. For the purposes of this documentation we will assume that the table is called `table_b` and that the table location is `s3://some_path/table_b`.
+TODO: what do we need to set up when we create this table programatically for everything to be registered correctly for read usage in Hive?

Review comment:
       Also, `engine.hive.enabled` is the table property. When a table has that property, it will be created with the storage handler. I set the property on my table from Spark and it worked.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org