[jira] [Created] (MADLIB-945) Investigate potential matrix memory leak

["Frank McQuillan (JIRA)" <ji...@apache.org>]

Frank McQuillan created MADLIB-945:
--------------------------------------

             Summary: Investigate potential matrix memory leak
                 Key: MADLIB-945
                 URL: https://issues.apache.org/jira/browse/MADLIB-945
             Project: Apache MADlib
          Issue Type: Bug
          Components: Module: Utilities
            Reporter: Frank McQuillan


Opened on behalf of Gautam Muralidhar:

Hi Guys,

So, I carried out some analysis for the memory bottlenecks last evening on the single node GPDB Sandbox VM . I stripped down the MCMC iteration code to a single table deletion and creation. Basically, I just drop a madlib matrix_product result table and create it again and repeat this drop and create for 3,000 iterations. Please see the attached code and look at the function mcmc.MemTest (you can focus on the loop for ite in range(0,num_iter):).

What I noticed was that the first 400 or so iterations go through quickly, but then it slows down immensely. For instance, I started a 3,000 iteration run last evening at around 6 PM and it was still running this morning at 10 AM! Now, this coupled with a couple of other operations needed for MCMC causes the query to error out with a 'failed to allocate memory' error.

Questions:

1. By any chance, is the MADlib matrix operation not freeing memory completely?

2. What overhead is introduced by repeated drop/create table?

3. I also observe that the MADlib matrix operations produce a lot of logging on the console. Can we disable this? Do you think this could be a problem, though, console write should be I/O. It will certainly slow down operations, though.

4. Lastly, what best practices do you have for creating tables in a loop? Do you guys do something like this in any of the current iterative algorithms in MADlib?

Lastly, this might vanish on the cluster.

Best,
Gautam



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)