You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "NeQuissimus (via GitHub)" <gi...@apache.org> on 2023/03/07 18:50:05 UTC
[GitHub] [iceberg] NeQuissimus opened a new issue, #7034: Hive ping functionality seems to leak threads
NeQuissimus opened a new issue, #7034:
URL: https://github.com/apache/iceberg/issues/7034
### Apache Iceberg version
1.1.0 (latest release)
### Query engine
Hive
### Please describe the bug 🐞
# Context
For context, this conversation started on Slack but we decided to move it here for better visibility.
cc @SinghAsDev
I believe this to be related to https://github.com/apache/iceberg/pull/5036
# Code
We have code using the API directly, no Spark, no Trickle etc.
The code looks a little something like this and is driven by a Kafka consumer:
```scala
// This is an object we keep around
def catalog: HiveCatalog
// for each message from Kafka, do the following
val table: Table = catalog.loadTable()
val transaction: Transaction = table.newTransaction()
val append = transaction.newFastAppend()
// add all data files from the Kafka message to `append`
...
transaction.commitTransaction()
```
# Observations
With Iceberg 0.14 (and 1.0.0, but we have not tested this extensively; enough to say the issue is not present there), we have a pretty steady state of threads running. Fetching a heap dump gives us maybe 150 threads across everything inside our application.
Once we update to Iceberg 1.1.0, we not only see the number of threads steadily increasing but also increasing with no obvious bound. (After a few hours, we see about 30,000 extra threads :D)
All of these threads are named `iceberg-hive-lock-heartbeat-0`, which is why I was looking at #5036 immediately and it is also a new change in Iceberg 1.1.0.
My understanding is that the `Transaction` essentially relates back to `HiveTableOperations.doCommit`.
I do not see anything in `HiveTableOperations` shutting down the scheduler for the Hive pings. But I am not sure whether that would even be necessary.
There are no new `close()` methods I could find on any of the objects we create either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Hive ping functionality seems to leak threads [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7034:
URL: https://github.com/apache/iceberg/issues/7034#issuecomment-1865301176
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #7034: Hive ping functionality seems to leak threads
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #7034:
URL: https://github.com/apache/iceberg/issues/7034#issuecomment-1512427785
> This might be fixed with : #6648 by @pvary . I see in [newest code](https://github.com/apache/iceberg/blob/master/hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java#L174) the code calls shutdown on exitingScheduledExecutorService. But in [earlier code](https://github.com/apache/iceberg/blob/apache-iceberg-1.1.0/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L383), it does not. Although I havent walked through it to be 100% sure.
This was the exact issue with the Spark test which prompted me to add the `cancel` call to the release of the locks. Otherwise the Spark tests were failing with resource problems.
The issue was highlighted by my change since the Locking change created threads on every commit instead on every TableOperation creation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #7034: Hive ping functionality seems to leak threads
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #7034:
URL: https://github.com/apache/iceberg/issues/7034#issuecomment-1460897515
I'm pretty sure that would fix it as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #7034: Hive ping functionality seems to leak threads
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on issue #7034:
URL: https://github.com/apache/iceberg/issues/7034#issuecomment-1502161935
It sounds like this should be a threadpool that is owned by the catalog, not a thread per table.
@pvary, can you confirm whether this is working in 1.2 and what we can do to avoid needing to manage resources per table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #7034: Hive ping functionality seems to leak threads
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #7034:
URL: https://github.com/apache/iceberg/issues/7034#issuecomment-1462998042
This might be fixed with : https://github.com/apache/iceberg/pull/6648/files by @pvary . I see in [newest code](https://github.com/apache/iceberg/blob/master/hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java#L174) the code calls shutdown on exitingScheduledExecutorService. But in [earlier code](https://github.com/apache/iceberg/blob/apache-iceberg-1.1.0/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L383), it does not. Although I havent walked through it to be 100% sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Hive ping functionality seems to leak threads [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7034: Hive ping functionality seems to leak threads
URL: https://github.com/apache/iceberg/issues/7034
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #7034: Hive ping functionality seems to leak threads
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #7034:
URL: https://github.com/apache/iceberg/issues/7034#issuecomment-1459367381
Quick check, are you sure all your HiveTableOperations objects are being finalized? The SingleThreadedExecutor service should get shut down during the finalizer as long as there are no running tasks (which there shouldn't be after the commit is done)
```java
public static ExecutorService newSingleThreadExecutor() {
return new FinalizableDelegatedExecutorService
(new ThreadPoolExecutor(1, 1,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>()));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #7034: Hive ping functionality seems to leak threads
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #7034:
URL: https://github.com/apache/iceberg/issues/7034#issuecomment-1512432038
> It sounds like this should be a threadpool that is owned by the catalog, not a thread per table.
>
> @pvary, can you confirm whether this is working in 1.2 and what we can do to avoid needing to manage resources per table?
We need different threads for heartbeating different Hive locks. It was technically OK when we bound it to the HiveTableOperation, as there were no concurrent commits for a single operation, but putting it on the Catalog level could cause noisy neighbors problem when multiple tables are using the same thread pool.
The issue should not be present after #6648, as it binds the heartbeat thread to the Lock itself, which is the correct place for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] NeQuissimus commented on issue #7034: Hive ping functionality seems to leak threads
Posted by "NeQuissimus (via GitHub)" <gi...@apache.org>.
NeQuissimus commented on issue #7034:
URL: https://github.com/apache/iceberg/issues/7034#issuecomment-1460884642
I am going to see if caching `Table` objects makes a significant difference here. I wonder if creating and stopping these thread pools/threads is too expensive considering how often we create `Table` objects currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Hive ping functionality seems to leak threads [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7034:
URL: https://github.com/apache/iceberg/issues/7034#issuecomment-1876137904
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org