You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/08/01 10:32:05 UTC
[GitHub] [iceberg] Fokko opened a new pull request, #5405: Hive: Hadoop Path fails on s3 endpoint
Fokko opened a new pull request, #5405:
URL: https://github.com/apache/iceberg/pull/5405
While testing locally with a Hive metastore with a Minio backend
```
spark-sql> create database nyc;
22/08/01 08:15:19 ERROR SparkSQLDriver: Failed in [create database nyc]
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: s3://warehousenyc
at org.apache.hadoop.fs.Path.initialize(Path.java:263)
at org.apache.hadoop.fs.Path.<init>(Path.java:161)
at org.apache.hadoop.fs.Path.<init>(Path.java:119)
at org.apache.iceberg.hive.HiveCatalog.convertToDatabase(HiveCatalog.java:510)
at org.apache.iceberg.hive.HiveCatalog.lambda$createNamespace$8(HiveCatalog.java:264)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
at org.apache.iceberg.hive.HiveCatalog.createNamespace(HiveCatalog.java:263)
at org.apache.iceberg.spark.SparkCatalog.createNamespace(SparkCatalog.java:407)
at org.apache.spark.sql.execution.datasources.v2.CreateNamespaceExec.run(CreateNamespaceExec.scala:47)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:220)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:384)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:504)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:498)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:498)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:286)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: s3://warehousenyc
at java.base/java.net.URI.checkPath(URI.java:1940)
at java.base/java.net.URI.<init>(URI.java:757)
at org.apache.hadoop.fs.Path.initialize(Path.java:260)
... 66 more
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: s3://warehousenyc
at org.apache.hadoop.fs.Path.initialize(Path.java:263)
at org.apache.hadoop.fs.Path.<init>(Path.java:161)
at org.apache.hadoop.fs.Path.<init>(Path.java:119)
at org.apache.iceberg.hive.HiveCatalog.convertToDatabase(HiveCatalog.java:510)
at org.apache.iceberg.hive.HiveCatalog.lambda$createNamespace$8(HiveCatalog.java:264)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
at org.apache.iceberg.hive.HiveCatalog.createNamespace(HiveCatalog.java:263)
at org.apache.iceberg.spark.SparkCatalog.createNamespace(SparkCatalog.java:407)
at org.apache.spark.sql.execution.datasources.v2.CreateNamespaceExec.run(CreateNamespaceExec.scala:47)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:220)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:384)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:504)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:498)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:498)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:286)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: s3://warehousenyc
at java.base/java.net.URI.checkPath(URI.java:1940)
at java.base/java.net.URI.<init>(URI.java:757)
at org.apache.hadoop.fs.Path.initialize(Path.java:260)
... 66 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on a diff in pull request #5405: Hive: Hadoop Path fails on s3 endpoint
Posted by GitBox <gi...@apache.org>.
Fokko commented on code in PR #5405:
URL: https://github.com/apache/iceberg/pull/5405#discussion_r934555444
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java:
##########
@@ -482,18 +481,19 @@ protected String defaultWarehouseLocation(TableIdentifier tableIdentifier) {
throw new RuntimeException("Interrupted during commit", e);
}
- // Otherwise stick to the {WAREHOUSE_DIR}/{DB_NAME}.db/{TABLE_NAME} path
- String warehouseLocation = getWarehouseLocation();
- return String.format(
- "%s/%s.db/%s",
- warehouseLocation, tableIdentifier.namespace().levels()[0], tableIdentifier.name());
+ // Otherwise, stick to the {WAREHOUSE_DIR}/{DB_NAME}.db/{TABLE_NAME} path
+ String warehouseLocation = getWarehouseLocation(tableIdentifier.namespace().levels()[0]);
+ return String.format("%s/%s", warehouseLocation, tableIdentifier.name());
}
- private String getWarehouseLocation() {
+ private String getWarehouseLocation(String warehouseName) {
String warehouseLocation = conf.get(HiveConf.ConfVars.METASTOREWAREHOUSE.varname);
Preconditions.checkNotNull(
warehouseLocation, "Warehouse location is not set: hive.metastore.warehouse.dir=null");
- return warehouseLocation;
+ if (warehouseLocation.endsWith("/")) {
+ warehouseLocation = warehouseLocation.substring(0, warehouseLocation.length() - 1);
+ }
Review Comment:
Nice, wasn't aware of that one. Updated it right away, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #5405: Hive: Hadoop Path fails on s3 endpoint
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5405:
URL: https://github.com/apache/iceberg/pull/5405#discussion_r937010778
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java:
##########
@@ -482,18 +481,17 @@ protected String defaultWarehouseLocation(TableIdentifier tableIdentifier) {
throw new RuntimeException("Interrupted during commit", e);
}
- // Otherwise stick to the {WAREHOUSE_DIR}/{DB_NAME}.db/{TABLE_NAME} path
- String warehouseLocation = getWarehouseLocation();
- return String.format(
- "%s/%s.db/%s",
- warehouseLocation, tableIdentifier.namespace().levels()[0], tableIdentifier.name());
+ // Otherwise, stick to the {WAREHOUSE_DIR}/{DB_NAME}.db/{TABLE_NAME} path
+ String databaseLocation = getDatabaseLocation(tableIdentifier.namespace().levels()[0]);
+ return String.format("%s/%s", databaseLocation, tableIdentifier.name());
}
- private String getWarehouseLocation() {
+ private String getDatabaseLocation(String databaseName) {
Review Comment:
Minor: We avoid using `get` in method names because it doesn't add anything useful. Usually, you can omit it or replace with a more specific verb, like "create" or "find". The action here is implied so I'd just omit "get".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #5405: Hive: Hadoop Path fails on s3 endpoint
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #5405:
URL: https://github.com/apache/iceberg/pull/5405#discussion_r934487399
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java:
##########
@@ -482,18 +481,16 @@ protected String defaultWarehouseLocation(TableIdentifier tableIdentifier) {
throw new RuntimeException("Interrupted during commit", e);
}
- // Otherwise stick to the {WAREHOUSE_DIR}/{DB_NAME}.db/{TABLE_NAME} path
- String warehouseLocation = getWarehouseLocation();
- return String.format(
- "%s/%s.db/%s",
- warehouseLocation, tableIdentifier.namespace().levels()[0], tableIdentifier.name());
+ // Otherwise, stick to the {WAREHOUSE_DIR}/{DB_NAME}.db/{TABLE_NAME} path
+ String warehouseLocation = getDatabaseLocation(tableIdentifier.namespace().levels()[0]);
+ return String.format("%s/%s", warehouseLocation, tableIdentifier.name());
}
- private String getWarehouseLocation() {
+ private String getDatabaseLocation(String warehouseName) {
Review Comment:
warehouseName -> databaseName
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] singhpk234 commented on a diff in pull request #5405: Hive: Hadoop Path fails on s3 endpoint
Posted by GitBox <gi...@apache.org>.
singhpk234 commented on code in PR #5405:
URL: https://github.com/apache/iceberg/pull/5405#discussion_r934530671
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java:
##########
@@ -482,18 +481,19 @@ protected String defaultWarehouseLocation(TableIdentifier tableIdentifier) {
throw new RuntimeException("Interrupted during commit", e);
}
- // Otherwise stick to the {WAREHOUSE_DIR}/{DB_NAME}.db/{TABLE_NAME} path
- String warehouseLocation = getWarehouseLocation();
- return String.format(
- "%s/%s.db/%s",
- warehouseLocation, tableIdentifier.namespace().levels()[0], tableIdentifier.name());
+ // Otherwise, stick to the {WAREHOUSE_DIR}/{DB_NAME}.db/{TABLE_NAME} path
+ String warehouseLocation = getWarehouseLocation(tableIdentifier.namespace().levels()[0]);
+ return String.format("%s/%s", warehouseLocation, tableIdentifier.name());
}
- private String getWarehouseLocation() {
+ private String getWarehouseLocation(String warehouseName) {
String warehouseLocation = conf.get(HiveConf.ConfVars.METASTOREWAREHOUSE.varname);
Preconditions.checkNotNull(
warehouseLocation, "Warehouse location is not set: hive.metastore.warehouse.dir=null");
- return warehouseLocation;
+ if (warehouseLocation.endsWith("/")) {
+ warehouseLocation = warehouseLocation.substring(0, warehouseLocation.length() - 1);
+ }
Review Comment:
[minor] can use [LocationUtil#stripTrailingSlash](https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/util/LocationUtil.java#L26) .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #5405: Hive: Hadoop Path fails on s3 endpoint
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #5405:
URL: https://github.com/apache/iceberg/pull/5405#issuecomment-1201162112
Would using a different warehouse uri help here?
If we absolute need this change, could we add a test case so there will be no regressions later?
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #5405: Hive: Hadoop Path fails on s3 endpoint
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #5405:
URL: https://github.com/apache/iceberg/pull/5405#issuecomment-1202031301
@Fokko: It seems, I wasn't clear enough in my review. Sorry for that. The path, which ends with `.db` is a `databaseLocation`, so we should use `getDatabaseLocation` and `databaseName` respectively.
Thanks, Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on a diff in pull request #5405: Hive: Hadoop Path fails on s3 endpoint
Posted by GitBox <gi...@apache.org>.
Fokko commented on code in PR #5405:
URL: https://github.com/apache/iceberg/pull/5405#discussion_r934507038
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java:
##########
@@ -482,18 +481,16 @@ protected String defaultWarehouseLocation(TableIdentifier tableIdentifier) {
throw new RuntimeException("Interrupted during commit", e);
}
- // Otherwise stick to the {WAREHOUSE_DIR}/{DB_NAME}.db/{TABLE_NAME} path
- String warehouseLocation = getWarehouseLocation();
- return String.format(
- "%s/%s.db/%s",
- warehouseLocation, tableIdentifier.namespace().levels()[0], tableIdentifier.name());
+ // Otherwise, stick to the {WAREHOUSE_DIR}/{DB_NAME}.db/{TABLE_NAME} path
+ String warehouseLocation = getDatabaseLocation(tableIdentifier.namespace().levels()[0]);
+ return String.format("%s/%s", warehouseLocation, tableIdentifier.name());
}
- private String getWarehouseLocation() {
+ private String getDatabaseLocation(String warehouseName) {
String warehouseLocation = conf.get(HiveConf.ConfVars.METASTOREWAREHOUSE.varname);
Preconditions.checkNotNull(
warehouseLocation, "Warehouse location is not set: hive.metastore.warehouse.dir=null");
- return warehouseLocation;
+ return String.format("%s/%s.db", warehouseLocation, warehouseName);
Review Comment:
Thanks for the review, much appreciated. I believe that the trailing slash is already trimmer, but I've added logic to remove the `/` at the end when present, including a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #5405: Hive: Hadoop Path fails on s3 endpoint
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #5405:
URL: https://github.com/apache/iceberg/pull/5405#issuecomment-1205104504
Thanks @Fokko for the fix and Ryan for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #5405: Hive: Hadoop Path fails on s3 endpoint
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #5405:
URL: https://github.com/apache/iceberg/pull/5405#issuecomment-1204333933
@Fokko, these changes look fine to me. I'll defer to @pvary to merge when he's had a chance to review.
I'm not sure that I understand how this fixes the initial failure case though. Was it just that the last `/` was missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary merged pull request #5405: Hive: Hadoop Path fails on s3 endpoint
Posted by GitBox <gi...@apache.org>.
pvary merged PR #5405:
URL: https://github.com/apache/iceberg/pull/5405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #5405: Hive: Hadoop Path fails on s3 endpoint
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #5405:
URL: https://github.com/apache/iceberg/pull/5405#discussion_r934491312
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java:
##########
@@ -482,18 +481,16 @@ protected String defaultWarehouseLocation(TableIdentifier tableIdentifier) {
throw new RuntimeException("Interrupted during commit", e);
}
- // Otherwise stick to the {WAREHOUSE_DIR}/{DB_NAME}.db/{TABLE_NAME} path
- String warehouseLocation = getWarehouseLocation();
- return String.format(
- "%s/%s.db/%s",
- warehouseLocation, tableIdentifier.namespace().levels()[0], tableIdentifier.name());
+ // Otherwise, stick to the {WAREHOUSE_DIR}/{DB_NAME}.db/{TABLE_NAME} path
+ String warehouseLocation = getDatabaseLocation(tableIdentifier.namespace().levels()[0]);
+ return String.format("%s/%s", warehouseLocation, tableIdentifier.name());
}
- private String getWarehouseLocation() {
+ private String getDatabaseLocation(String warehouseName) {
String warehouseLocation = conf.get(HiveConf.ConfVars.METASTOREWAREHOUSE.varname);
Preconditions.checkNotNull(
warehouseLocation, "Warehouse location is not set: hive.metastore.warehouse.dir=null");
- return warehouseLocation;
+ return String.format("%s/%s.db", warehouseLocation, warehouseName);
Review Comment:
There might be some unexpected differences between the old path and the new path. One such thing I can think of is if the configuration contains a `/` character at the end of the string. There might be some other differences also.
We should make sure that we minimise/remove these differences.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on pull request #5405: Hive: Hadoop Path fails on s3 endpoint
Posted by GitBox <gi...@apache.org>.
Fokko commented on PR #5405:
URL: https://github.com/apache/iceberg/pull/5405#issuecomment-1204340849
@rdblue Thanks for reviewing. It is when the location is somewhere without a child. The trailing `/` is removed from hive warehouse dir. This way you can never use a base url because it thinks it is a relative path:
```
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.3.0
/_/
Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 18.0.2)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import org.apache.hadoop.fs.Path
import org.apache.hadoop.fs.Path
scala> new Path("s3://warehouse", "nyc")
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: s3://warehousenyc
at org.apache.hadoop.fs.Path.initialize(Path.java:263)
at org.apache.hadoop.fs.Path.<init>(Path.java:161)
at org.apache.hadoop.fs.Path.<init>(Path.java:119)
... 43 elided
Caused by: java.net.URISyntaxException: Relative path in absolute URI: s3://warehousenyc
at java.base/java.net.URI.checkPath(URI.java:1998)
at java.base/java.net.URI.<init>(URI.java:780)
at org.apache.hadoop.fs.Path.initialize(Path.java:260)
... 45 more
scala> new Path("s3://warehouse/", "nyc")
res2: org.apache.hadoop.fs.Path = s3://warehouse/nyc
scala> new Path("s3://warehouse/abc", "nyc")
res3: org.apache.hadoop.fs.Path = s3://warehouse/abc/nyc
```
I think we should just remove the Hadoop Path here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on pull request #5405: Hive: Hadoop Path fails on s3 endpoint
Posted by GitBox <gi...@apache.org>.
Fokko commented on PR #5405:
URL: https://github.com/apache/iceberg/pull/5405#issuecomment-1202128835
Sorry about that @pvary. I've just pushed a fix. Don't forget, that naming things is one of the hardest things in computer science :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #5405: Hive: Hadoop Path fails on s3 endpoint
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #5405:
URL: https://github.com/apache/iceberg/pull/5405#discussion_r934486963
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java:
##########
@@ -482,18 +481,16 @@ protected String defaultWarehouseLocation(TableIdentifier tableIdentifier) {
throw new RuntimeException("Interrupted during commit", e);
}
- // Otherwise stick to the {WAREHOUSE_DIR}/{DB_NAME}.db/{TABLE_NAME} path
- String warehouseLocation = getWarehouseLocation();
- return String.format(
- "%s/%s.db/%s",
- warehouseLocation, tableIdentifier.namespace().levels()[0], tableIdentifier.name());
+ // Otherwise, stick to the {WAREHOUSE_DIR}/{DB_NAME}.db/{TABLE_NAME} path
+ String warehouseLocation = getDatabaseLocation(tableIdentifier.namespace().levels()[0]);
Review Comment:
The variable name should be `databaseLocation` instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org