You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@heron.apache.org by GitBox <gi...@apache.org> on 2022/02/02 18:39:37 UTC
[GitHub] [incubator-heron] nicknezis opened a new issue #3769: Uploader implementations may leak storage when resubmitting the same analytic multiple times
nicknezis opened a new issue #3769:
URL: https://github.com/apache/incubator-heron/issues/3769
**Describe the bug**
The Uploader logic in Heron adds a Random Long to the filename that causes memory leak in Bookkeeper.
The Uploader filename generation logic is [here](https://github.com/apache/incubator-heron/blob/abb2767e3df3ca6eba009f46efe1f1e83695617a/heron/spi/src/java/org/apache/heron/spi/utils/UploaderUtils.java#L62).
And the Dlog config that disables time and size based retention is [here](https://github.com/apache/incubator-heron/blob/abb2767e3df3ca6eba009f46efe1f1e83695617a/heron/uploaders/src/java/org/apache/heron/uploader/dlog/DLUploader.java#L122-L128).
Eventually the Bookkeeper log runs out of space and no more topologies can be submitted.
The proposed fix is to remove the Random.long that is being added to the filename.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@heron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-heron] windhamwong commented on issue #3769: Uploader implementations may leak storage when resubmitting the same analytic multiple times
Posted by GitBox <gi...@apache.org>.
windhamwong commented on issue #3769:
URL: https://github.com/apache/incubator-heron/issues/3769#issuecomment-1042515662
Is that possible to, either defined by user, or set it with timestamp instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@heron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-heron] nicknezis commented on issue #3769: Uploader implementations may leak storage when resubmitting the same analytic multiple times
Posted by GitBox <gi...@apache.org>.
nicknezis commented on issue #3769:
URL: https://github.com/apache/incubator-heron/issues/3769#issuecomment-1033408181
So my suggestion is to remove the Random.long all together. I'm not sure why this was added, but we don't have any ability to "roll-back" to the previous uploads. So having an endless series of entries in Bookkeeper doesn't make sense. If I resubmit the same topology with name, role, version coordinates, I would expect it to replace the previous entry in Bookkeeper (or whatever upload mechanism is used). For example when developing and testing an analytic job.
If the desire is to run different binaries, one should distinguish the name, role and/or version to create a unique coordinate in the Upload/Download mechanism.
We will have to test the Upload functionality to ensure that removing the Random.long doesn't break anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@heron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-heron] surahman commented on issue #3769: Uploader implementations may leak storage when resubmitting the same analytic multiple times
Posted by GitBox <gi...@apache.org>.
surahman commented on issue #3769:
URL: https://github.com/apache/incubator-heron/issues/3769#issuecomment-1028313605
Would using [RandomStringUtils#random](https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/RandomStringUtils.html) with a character count of 11 fix the issue? Or is this a string length related issue?
If my math is correct it should generate 26 `uppercase` + 26 `lowercase` + 10 `digits` permute 11 `count` = 62 p 11 = 390,164,706,723,052,800 potential strings. That is more options than `Random#long` provides.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@heron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org