You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Evgeny <sm...@mail.ru.INVALID> on 2017/06/29 23:23:38 UTC
IncomingMessageEnvelope - memory leak
Hello. Heap size of running samza container is about 7 Gb and there are a huge numbers of IncomingMessageEnvelope as unreachable objects in thread dump. What's the reason of this? It's just a samza task with empty process method.
Re: Re[2]: IncomingMessageEnvelope - memory leak
Posted by Jacob Maes <ja...@gmail.com>.
On Fri, Jun 30, 2017 at 7:25 AM, Evgeny <sm...@mail.ru> wrote:
> Sorry didn't understand this question: "Is the container able to keep up
> with its inputs?" What do you mean keep up with its inputs?
>
Apologies for the vague phrasing. I meant to ask if the input topics have a
high volume of messages. If that's the case, and if the job cannot process
those messages fast enough to keep up, then it often fills the buffers I
mentioned in the previous message.
Re[2]: IncomingMessageEnvelope - memory leak
Posted by Evgeny <sm...@mail.ru.INVALID>.
Hi. The input stream is kafka and samza is running on mesos using this framework https://github.com/Banno/samza-mesos
Samza version - 0.12. Sorry didn't understand this question: "Is the container able to keep up with its inputs?" What do you mean keep up with its inputs?
>Пятница, 30 июня 2017, 2:42 +03:00 от Jacob Maes <ja...@gmail.com>:
>
>Hey Evgeny,
>
>We haven't seen this with our jobs, but here are some thoughts and
>questions for you to consider.
>
>Is the container able to keep up with its inputs? The further behind, the
>more it will fill buffers.
>
>Samza has one buffer, which is controlled by configs
>< http://samza.apache.org/learn/documentation/0.13/jobs/configuration-table.html >
>systems.system-name.samza.fetch.threshold
>or
>systems.system-name.samza.fetch.threshold.bytes
>
>Newer kafka consumer clients have their own NIO buffers controlled by a
>`fetch.max.bytes` config, though that wouldn't explain why you see so many
>IncomingMessageEnvelopes.
>
>What version of Samza is the job using?
>
>Anything unique about the inputs or configuration?
>
>-Jake
>
>On Thu, Jun 29, 2017 at 4:23 PM, Evgeny < smyvin@mail.ru.invalid > wrote:
>
>>
>> Hello. Heap size of running samza container is about 7 Gb and there are a
>> huge numbers of IncomingMessageEnvelope as unreachable objects in thread
>> dump. What's the reason of this? It's just a samza task with empty process
>> method.
>>
>>
>>
>>
>>
Re: IncomingMessageEnvelope - memory leak
Posted by Jacob Maes <ja...@gmail.com>.
Hey Evgeny,
We haven't seen this with our jobs, but here are some thoughts and
questions for you to consider.
Is the container able to keep up with its inputs? The further behind, the
more it will fill buffers.
Samza has one buffer, which is controlled by configs
<http://samza.apache.org/learn/documentation/0.13/jobs/configuration-table.html>
systems.system-name.samza.fetch.threshold
or
systems.system-name.samza.fetch.threshold.bytes
Newer kafka consumer clients have their own NIO buffers controlled by a
`fetch.max.bytes` config, though that wouldn't explain why you see so many
IncomingMessageEnvelopes.
What version of Samza is the job using?
Anything unique about the inputs or configuration?
-Jake
On Thu, Jun 29, 2017 at 4:23 PM, Evgeny <sm...@mail.ru.invalid> wrote:
>
> Hello. Heap size of running samza container is about 7 Gb and there are a
> huge numbers of IncomingMessageEnvelope as unreachable objects in thread
> dump. What's the reason of this? It's just a samza task with empty process
> method.
>
>
>
>
>