[jira] [Closed] (MADLIB-1372) MADlib Keras operations create too many threads

["Orhan Kislal (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MADLIB-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Orhan Kislal closed MADLIB-1372.
--------------------------------
    Resolution: Fixed

> MADlib Keras operations create too many threads
> -----------------------------------------------
>
>                 Key: MADLIB-1372
>                 URL: https://issues.apache.org/jira/browse/MADLIB-1372
>             Project: Apache MADlib
>          Issue Type: Bug
>          Components: Deep Learning
>            Reporter: Orhan Kislal
>            Priority: Major
>             Fix For: v1.17
>
>
> We noticed that madlib keras operations that call keras functions create a number of threads every single time they are called. However, some of these threads are not cleaned up at the end of the function. If the number of iterations is very high, the thread count keeps increasing and eventually hits the system default limit (1024).
> Here is the error message from the log
> {code}
> what(): Resource temporarily unavailable",,,,,,,,"SysLoggerMain","syslogger.c",618,
> {code}
> We tried keeping the session at the end of the function (saved the session info in SD) and reuse it for the next iteration but that didn't help with this issue.
> It is possible to increase this limit by editing /etc/security/limits.conf and the files in /etc/security/limits.d/. This requires a restart of the system and the database to take effect. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)