You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/09/24 01:33:08 UTC
[GitHub] [iceberg] waterlx opened a new issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
waterlx opened a new issue #1502:
URL: https://github.com/apache/iceberg/issues/1502
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] waterlx commented on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
waterlx commented on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-716146541
> I think that the lifecycle of a catalog should be tied to the lifecycle of a SQL session, which addresses the problem because the catalog is not closed unless the session is.
>
> I'm not aware of a catalog cache anywhere. Is that something you have added?
@rdblue I think what Junjie talked about is HiveCatalogs. It calles close() of HiveCatalog when the cache entry is removed. And the corresponding HiveClientPool is closed as a result.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-805370516
Yeah, I think you're right. In 2.4, Spark doesn't maintain the catalogs. The cache assumes that Spark is maintaining the references. We should use `softValues` for Spark 3 and later, but not for 2.4. Would you like to open a PR to fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-717598760
Thanks, @waterlx! That explains it. I agree, we should remove the expiration from that catalog cache.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] waterlx commented on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
waterlx commented on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-698357438
@rdblue Could you please share your idea about that? if I understand that correctly, you would like to make HiveClientPool (created by HiveCatalog) unavailable to use right after HiveCatalog object is closed or garbage-collected. So HiveClientPool is supposed to have the same life cycle as HiveCatalog?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-698635234
Yeah, I think that the lifecycle of the client pool should be the same as the catalog. We want to reuse a client pool to avoid opening too many connections to the Hive MetaStore the catalog is configured to use. The most reasonable place to do that is with the catalog.
We could separate them, so the catalog is passed a client pool, but it isn't clear to me what that would accomplish: why would we create identical catalogs that use the same external pool? That would also be difficult to manage because we would have to make sure that the pool's connections are made to the same HMS that the catalog is configured to use.
If it doesn't make sense to have an external client pool and we want the client pool to be reused and have a long lifecycle, then it makes sense to keep the catalog around to manage that lifecycle. Sounds like the problem here is that code is creating and discarding a catalog before the tables that were created by it are no longer needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] springle commented on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
springle commented on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-797557395
Hi - I'm not sure if this should be a separate issue, but I still see this error with some frequency while using Spark 2.4.4 and writing to an existing Iceberg table. It's non-deterministic, but it only seems to happen for longer writes (30+ minutes). I also see this TODO which seems related (https://github.com/apache/iceberg/blob/master/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java#L522), but I'm not too familiar with Iceberg internals yet.
```
Py4JJavaError: An error occurred while calling o7137.save.
: org.apache.spark.SparkException: Writing job aborted.
at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec.doExecute(WriteToDataSourceV2Exec.scala:92)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:260)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Cannot get a client from a closed pool
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkState(Preconditions.java:508)
at org.apache.iceberg.hive.ClientPool.get(ClientPool.java:110)
at org.apache.iceberg.hive.ClientPool.run(ClientPool.java:52)
at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:130)
at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:86)
at org.apache.iceberg.SnapshotProducer.refresh(SnapshotProducer.java:258)
at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:155)
at org.apache.iceberg.BaseReplacePartitions.apply(BaseReplacePartitions.java:26)
at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:276)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:213)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:197)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:189)
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:275)
at org.apache.iceberg.BaseReplacePartitions.commit(BaseReplacePartitions.java:26)
at org.apache.iceberg.spark.source.Writer.commitOperation(Writer.java:172)
at org.apache.iceberg.spark.source.Writer.replacePartitions(Writer.java:198)
at org.apache.iceberg.spark.source.Writer.commit(Writer.java:148)
at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec.doExecute(WriteToDataSourceV2Exec.scala:76)
... 27 more
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] springle commented on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
springle commented on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-797603522
From what I can understand, the child `Table` object is created with the catalog before [initializing the Spark writer](https://github.com/apache/iceberg/blob/master/spark2/src/main/java/org/apache/iceberg/spark/source/IcebergSource.java#L93), and then the reference to the catalog is lost (and sometimes GC'd, closing the connection pool in Hive's case). Who should own this reference - the `IcebergSource`, the `Writer`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] springle commented on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
springle commented on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-803592326
Hi @rdblue - this is with Iceberg 11 and Spark 2.4, and I think the catalog in the link above ends up getting loaded from [here](https://github.com/apache/iceberg/blob/629f7ad7c5fbc16da1e0752f5d170889d4e761b4/spark2/src/main/java/org/apache/iceberg/spark/source/CustomCatalogs.java#L89). I see what you're saying about the catalog cache being tied to a Spark session, but maybe using `softValues` lets it get cleaned up prematurely?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-799723563
@springle, I think caching the client pool could definitely help. Another thing is that we may need to take a look at the lifecycle of the catalog you're using. They're supposed to be tied to the lifecycle of the Spark session so the pool shouldn't be closed. What version of Iceberg are you using?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] springle edited a comment on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
springle edited a comment on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-797603522
From what I can understand, the child `Table` object is created with the catalog before [initializing the Spark writer](https://github.com/apache/iceberg/blob/master/spark2/src/main/java/org/apache/iceberg/spark/source/IcebergSource.java#L93), and then the reference to the catalog is lost (and sometimes GC'd, closing the connection pool in Hive's case). Who should own this reference - the `IcebergSource`, the `Writer`?
EDIT: nevermind, looks like https://github.com/apache/iceberg/pull/2325 will resolve this!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] waterlx edited a comment on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
waterlx edited a comment on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-716146541
> I think that the lifecycle of a catalog should be tied to the lifecycle of a SQL session, which addresses the problem because the catalog is not closed unless the session is.
>
> I'm not aware of a catalog cache anywhere. Is that something you have added?
@rdblue I think what Junjie talked about is HiveCatalogs. It calles close() of HiveCatalog when the cache entry is removed. T The corresponding HiveClientPool is closed as a result.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] waterlx edited a comment on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
waterlx edited a comment on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-698357438
@rdblue Could you please share your idea about that? if I understand that correctly, you would like to make HiveClientPool (created by HiveCatalog) unavailable to use right after HiveCatalog is closed or garbage-collected. So HiveClientPool is supposed to have the same life cycle as HiveCatalog?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue closed issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
rdblue closed issue #1502:
URL: https://github.com/apache/iceberg/issues/1502
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] waterlx edited a comment on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
waterlx edited a comment on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-698357438
@rdblue Could you please share your idea about that? if I understand that correctly, you would like to make HiveClientPool (created by HiveCatalog) unavailable to use right after HiveCatalog is closed or garbage-collected. So HiveClientPool is supposed to have the same life cycle as HiveCatalog?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] chenjunjiedada commented on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-714364533
This also happens when the catalog is expired by catalog cache but the table is still in use. Currently, we expire the catalog after 10 minutes of last access. While actions like a rewrite or merge into might spend more time than it. @aokolnychyi, Did you encounter this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] waterlx commented on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
waterlx commented on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-698357438
@rdblue Could you please share your idea about that? if I understand that correctly, you would like to make HiveClientPool (created by HiveCatalog) unavailable to use right after HiveCatalog object is closed or garbage-collected. So HiveClientPool is supposed to have the same life cycle as HiveCatalog?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-698635234
Yeah, I think that the lifecycle of the client pool should be the same as the catalog. We want to reuse a client pool to avoid opening too many connections to the Hive MetaStore the catalog is configured to use. The most reasonable place to do that is with the catalog.
We could separate them, so the catalog is passed a client pool, but it isn't clear to me what that would accomplish: why would we create identical catalogs that use the same external pool? That would also be difficult to manage because we would have to make sure that the pool's connections are made to the same HMS that the catalog is configured to use.
If it doesn't make sense to have an external client pool and we want the client pool to be reused and have a long lifecycle, then it makes sense to keep the catalog around to manage that lifecycle. Sounds like the problem here is that code is creating and discarding a catalog before the tables that were created by it are no longer needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #1502: HiveClientPool closed by HiveCatalog blocks table operations
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #1502:
URL: https://github.com/apache/iceberg/issues/1502#issuecomment-715466468
I think that the lifecycle of a catalog should be tied to the lifecycle of a SQL session, which addresses the problem because the catalog is not closed unless the session is.
I'm not aware of a catalog cache anywhere. Is that something you have added?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org