[GitHub] [iceberg] springle commented on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations

[GitBox <gi...@apache.org>]

springle commented on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-797557395


   Hi - I'm not sure if this should be a separate issue,  but I still see this error with some frequency while using Spark 2.4.4 and writing to an existing Iceberg table. It's non-deterministic, but it only seems to happen for longer writes (30+ minutes). I also see this TODO which seems related (https://github.com/apache/iceberg/blob/master/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java#L522), but I'm not too familiar with Iceberg internals yet.
   
   ```
   Py4JJavaError: An error occurred while calling o7137.save.
   : org.apache.spark.SparkException: Writing job aborted.
   	at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec.doExecute(WriteToDataSourceV2Exec.scala:92)
   	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
   	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
   	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
   	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
   	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
   	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
   	at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
   	at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
   	at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
   	at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
   	at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
   	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
   	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
   	at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
   	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:260)
   	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
   	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
   	at py4j.Gateway.invoke(Gateway.java:282)
   	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
   	at py4j.commands.CallCommand.execute(CallCommand.java:79)
   	at py4j.GatewayConnection.run(GatewayConnection.java:238)
   	at java.lang.Thread.run(Thread.java:748)
   Caused by: java.lang.IllegalStateException: Cannot get a client from a closed pool
   	at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkState(Preconditions.java:508)
   	at org.apache.iceberg.hive.ClientPool.get(ClientPool.java:110)
   	at org.apache.iceberg.hive.ClientPool.run(ClientPool.java:52)
   	at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:130)
   	at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:86)
   	at org.apache.iceberg.SnapshotProducer.refresh(SnapshotProducer.java:258)
   	at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:155)
   	at org.apache.iceberg.BaseReplacePartitions.apply(BaseReplacePartitions.java:26)
   	at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:276)
   	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
   	at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:213)
   	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:197)
   	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:189)
   	at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:275)
   	at org.apache.iceberg.BaseReplacePartitions.commit(BaseReplacePartitions.java:26)
   	at org.apache.iceberg.spark.source.Writer.commitOperation(Writer.java:172)
   	at org.apache.iceberg.spark.source.Writer.replacePartitions(Writer.java:198)
   	at org.apache.iceberg.spark.source.Writer.commit(Writer.java:148)
   	at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec.doExecute(WriteToDataSourceV2Exec.scala:76)
   	... 27 more
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org