[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #239: Update iceberg-spark to use HiveTables

[GitBox <gi...@apache.org>]

rdblue commented on a change in pull request #239: Update iceberg-spark to use HiveTables
URL: https://github.com/apache/incubator-iceberg/pull/239#discussion_r298416948
 
 

 ##########
 File path: spark/src/main/java/org/apache/iceberg/spark/source/IcebergSource.java
 ##########
 @@ -96,13 +97,16 @@ public StreamWriter createStreamWriter(String runId, StructType dsStruct,
   }
 
   protected Table findTable(DataSourceOptions options, Configuration conf) {
-    Optional<String> location = options.get("path");
-    Preconditions.checkArgument(location.isPresent(),
-        "Cannot open table without a location: path is not set");
-
-    HadoopTables tables = new HadoopTables(conf);
-
-    return tables.load(location.get());
+    Optional<String> path = options.get("path");
+    Preconditions.checkArgument(path.isPresent(), "Cannot open table: path is not set");
+
+    if (path.get().contains("/")) {
+      HadoopTables tables = new HadoopTables(conf);
+      return tables.load(path.get());
+    } else {
+      HiveTables tables = new HiveTables(conf);
 
 Review comment:
   `HiveTables` creates a connection pool that we will need to close. Maybe we don't need to worry about this for now because Spark 3.0 will use a catalog with a more reasonable life-cycle, but it seems like a bad idea to leak connections. It is also difficult to close this because these are instantiated every time.
   
   What about creating a static cache and getting a `HiveTables` instance based on the value of `hive.metastore.uris`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org