TaskMemoryManager. cleanUpAllAllocatedMemory -> Memory leaks ???

[Lei Wu <wu...@gmail.com>]

Dear all,

I'm reading source code of TaskMemoryManager.java and I got stuck in the
last function, that is cleanUpAllAllocatedMemory. What confuses me is the
comments for this function : "A non-zero return value can be used to detect
memory leaks".

And from Executor.scala where this function is called, the code is like
this :
val freedMemory = taskMemoryManager.cleanUpAllAllocatedMemory()
if(freedMemory > 0) {
  val errMsg = "Managed memory leak detected; ..."
}

My question is : this function cleanUpAllAllocatedMemory is used to clean
up all the allocated memory and pages, and computes the bytes that are
freed. Of course this value will be greater than 0 since we consumed that
many amount of memory. What's the logic of detecting memory leak by
checking this returned value is greater than 0?

Re: TaskMemoryManager. cleanUpAllAllocatedMemory -> Memory leaks ???

[Ted Yu <yu...@gmail.com>]

Please take a look at the design doc attached to SPARK-10000

The answer is on page 2 of that doc.

On Mon, Oct 12, 2015 at 8:55 AM, Ted Yu <yu...@gmail.com> wrote:

> Please note the block where cleanUpAllAllocatedMemory() is called:
>         } finally {
>           val freedMemory = taskMemoryManager.cleanUpAllAllocatedMemory()
>           if (freedMemory > 0) {
> I think the intention is that allocated memory should have been freed by
> the time we reach the finally block in normal program execution.
> Hence the check for the return value for memory leak.
>
> Experts may have better explanation.
>
> On Mon, Oct 12, 2015 at 12:28 AM, Lei Wu <wu...@gmail.com> wrote:
>
>> Dear all,
>>
>> I'm reading source code of TaskMemoryManager.java and I got stuck in the
>> last function, that is cleanUpAllAllocatedMemory. What confuses me is the
>> comments for this function : "A non-zero return value can be used to detect
>> memory leaks".
>>
>> And from Executor.scala where this function is called, the code is like
>> this :
>> val freedMemory = taskMemoryManager.cleanUpAllAllocatedMemory()
>> if(freedMemory > 0) {
>>   val errMsg = "Managed memory leak detected; ..."
>> }
>>
>> My question is : this function cleanUpAllAllocatedMemory is used to clean
>> up all the allocated memory and pages, and computes the bytes that are
>> freed. Of course this value will be greater than 0 since we consumed that
>> many amount of memory. What's the logic of detecting memory leak by
>> checking this returned value is greater than 0?
>>
>
>

Re: TaskMemoryManager. cleanUpAllAllocatedMemory -> Memory leaks ???

[Ted Yu <yu...@gmail.com>]

Please note the block where cleanUpAllAllocatedMemory() is called:
        } finally {
          val freedMemory = taskMemoryManager.cleanUpAllAllocatedMemory()
          if (freedMemory > 0) {
I think the intention is that allocated memory should have been freed by
the time we reach the finally block in normal program execution.
Hence the check for the return value for memory leak.

Experts may have better explanation.

On Mon, Oct 12, 2015 at 12:28 AM, Lei Wu <wu...@gmail.com> wrote:

> Dear all,
>
> I'm reading source code of TaskMemoryManager.java and I got stuck in the
> last function, that is cleanUpAllAllocatedMemory. What confuses me is the
> comments for this function : "A non-zero return value can be used to detect
> memory leaks".
>
> And from Executor.scala where this function is called, the code is like
> this :
> val freedMemory = taskMemoryManager.cleanUpAllAllocatedMemory()
> if(freedMemory > 0) {
>   val errMsg = "Managed memory leak detected; ..."
> }
>
> My question is : this function cleanUpAllAllocatedMemory is used to clean
> up all the allocated memory and pages, and computes the bytes that are
> freed. Of course this value will be greater than 0 since we consumed that
> many amount of memory. What's the logic of detecting memory leak by
> checking this returned value is greater than 0?
>