You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Sendoh <un...@gmail.com> on 2017/05/15 16:01:23 UTC

memory usage in task manager when run and cancel a job

Hi Flink users,

How does memory usage percentage changes when starting the first job, second
job and cancel a job which all use the same jar?

I found that when starting the first job, it uses much more memory than the
second job. 
The first job uses around 20 % and the second one may use only 5%.

When canceling one of the jobs, the memory usage doesn't decrease. It means
GC is not triggered immediately? But even after 1 day the memory is not
released. 

My doubt is from we have a usecase which I wonder this has memory leak. I'm
not sure the memory is not released due to the job itself, or sth else.

Best,

Sendoh



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/memory-usage-in-task-manager-when-run-and-cancel-a-job-tp13142.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: memory usage in task manager when run and cancel a job

Posted by Sendoh <un...@gmail.com>.
Could I also ask is this behavior the same as master?

I saw that when master uses more than 100% memory (starting a new job uses
35%, and master already uses 70%), ubuntu shuts down and restarts.

Best,

Sendoh



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/memory-usage-in-task-manager-when-job-runs-and-cancels-tp13142p13145.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: memory usage in task manager when run and cancel a job

Posted by Sendoh <un...@gmail.com>.
Hi Fabian,

Thank you for quick reply. I run the job in streaming environment. 
So I think in streaming env memory is allocated up to the configured amount
and never returned until Flink is shutdown as you said if I understand well.

Best,

Sendoh




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/memory-usage-in-task-manager-when-job-runs-and-cancels-tp13142p13144.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: memory usage in task manager when run and cancel a job

Posted by Fabian Hueske <fh...@gmail.com>.
Hi Sendoh,

are you running batch or streaming jobs?
In batch mode, workers allocate managed memory lazily by default.
Memory is allocated up to the configured amount and never returned until
Flink is shutdown.
So, this would not indicate a memory leak.

Best, Fabian

2017-05-15 18:01 GMT+02:00 Sendoh <un...@gmail.com>:

> Hi Flink users,
>
> How does memory usage percentage changes when starting the first job,
> second
> job and cancel a job which all use the same jar?
>
> I found that when starting the first job, it uses much more memory than the
> second job.
> The first job uses around 20 % and the second one may use only 5%.
>
> When canceling one of the jobs, the memory usage doesn't decrease. It means
> GC is not triggered immediately? But even after 1 day the memory is not
> released.
>
> My doubt is from we have a usecase which I wonder this has memory leak. I'm
> not sure the memory is not released due to the job itself, or sth else.
>
> Best,
>
> Sendoh
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/memory-usage-in-
> task-manager-when-run-and-cancel-a-job-tp13142.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>