You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/11/30 09:45:11 UTC
[GitHub] [kafka] lucasbru opened a new pull request, #12925: Revert "[KAFKA-14324] Upgrade RocksDB to 7.1.2 (#12809)"
lucasbru opened a new pull request, #12925:
URL: https://github.com/apache/kafka/pull/12925
This reverts commit 876c338a60f2d923568ff282ff499a14a9ab83f1.
The new version of RocksDB is leaking memory, to the point that with this change, a stream thread will only survive a few hours without running out of memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lucasbru commented on pull request #12925: Revert "[KAFKA-14324] Upgrade RocksDB to 7.1.2 (#12809)"
Posted by GitBox <gi...@apache.org>.
lucasbru commented on PR #12925:
URL: https://github.com/apache/kafka/pull/12925#issuecomment-1332159428
This was discovered using a long-running Streams test, but the tests are run on confluent infrastructure and are not publicly accessible. I would think the memory leak can be reproduced with every other long-running job and if not, this could also help us to identify the root cause of the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #12925: Revert "[KAFKA-14324] Upgrade RocksDB to 7.1.2 (#12809)"
Posted by GitBox <gi...@apache.org>.
cadonna commented on PR #12925:
URL: https://github.com/apache/kafka/pull/12925#issuecomment-1331889635
\cc @ableegoldman
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lucasbru closed pull request #12925: Revert "[KAFKA-14324] Upgrade RocksDB to 7.1.2 (#12809)"
Posted by GitBox <gi...@apache.org>.
lucasbru closed pull request #12925: Revert "[KAFKA-14324] Upgrade RocksDB to 7.1.2 (#12809)"
URL: https://github.com/apache/kafka/pull/12925
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lucasbru commented on pull request #12925: Revert "[KAFKA-14324] Upgrade RocksDB to 7.1.2 (#12809)"
Posted by GitBox <gi...@apache.org>.
lucasbru commented on PR #12925:
URL: https://github.com/apache/kafka/pull/12925#issuecomment-1333536861
@cadonna Could you retrigger this? It is just one configuration (JDK 8 and Scala 2.12) that has lots of tests timing out, so that looks more like a node issue to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #12925: Revert "[KAFKA-14324] Upgrade RocksDB to 7.1.2 (#12809)"
Posted by GitBox <gi...@apache.org>.
clolov commented on PR #12925:
URL: https://github.com/apache/kafka/pull/12925#issuecomment-1332014214
I understand that this is a fairly urgent change and that we should revert the commit if there is a memory leak, but I am interested in how was this discovered in case I start looking at other RocksDB versions to move to? For example, are there any accessible long-running tests somewhere which unearthed this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #12925: Revert "[KAFKA-14324] Upgrade RocksDB to 7.1.2 (#12809)"
Posted by GitBox <gi...@apache.org>.
cadonna commented on PR #12925:
URL: https://github.com/apache/kafka/pull/12925#issuecomment-1332539943
There is something fishy in the build failures. There many Streams failures and I saw the following:
```
java.lang.IllegalArgumentException: Cannot get result for failed query. Failure is STORE_EXCEPTION: class org.apache.kafka.streams.state.internals.RocksDBStore failed to handle query org.apache.kafka.streams.query.RangeQuery@38c5b853:
org.apache.kafka.streams.errors.InvalidStateStoreException: Store store is currently closed
at org.apache.kafka.streams.state.internals.RocksDBStore.validateStoreOpen(RocksDBStore.java:300)
at org.apache.kafka.streams.state.internals.RocksDBStore.all(RocksDBStore.java:443)
at org.apache.kafka.streams.state.internals.RocksDBStore.all(RocksDBStore.java:434)
at org.apache.kafka.streams.state.internals.StoreQueryUtils.runRangeQuery(StoreQueryUtils.java:192)
```
@lucasbru @clolov Could you also have a look at the build failures?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #12925: Revert "[KAFKA-14324] Upgrade RocksDB to 7.1.2 (#12809)"
Posted by GitBox <gi...@apache.org>.
cadonna commented on PR #12925:
URL: https://github.com/apache/kafka/pull/12925#issuecomment-1333730811
I've already retriggered it once yesterday but I can retrigger again now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lucasbru commented on pull request #12925: Revert "[KAFKA-14324] Upgrade RocksDB to 7.1.2 (#12809)"
Posted by GitBox <gi...@apache.org>.
lucasbru commented on PR #12925:
URL: https://github.com/apache/kafka/pull/12925#issuecomment-1333981153
Tests look fine, now.
However, in the meantime I have found the cause of the leak (or at least one) in trunk. We could consider waiting for this fix to go in and not revert this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org