[jira] Commented: (SOLR-1042) Memory leaks in DIH

["Noble Paul (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677658#action_12677658 ] 

Noble Paul commented on SOLR-1042:
----------------------------------

I guess #2 is a better solution. 

going forward we recommend making 'deltaImportQuery' a must for doing delta. Then this problem must go away

> Memory leaks in DIH
> -------------------
>
>                 Key: SOLR-1042
>                 URL: https://issues.apache.org/jira/browse/SOLR-1042
>             Project: Solr
>          Issue Type: Bug
>          Components: contrib - DataImportHandler
>    Affects Versions: 1.3
>            Reporter: Ryuuichi Kumai
>         Attachments: SOLR-1042.patch
>
>
> If delta-import is executed many times, the heap utilization grows up and finally OutOfMemoryError occurs.
> When delta-import is executed with SqlEntityProcessor, the instances of TemplateString cached in VariableResolverImpl#TEMPLATE_STRING#cache.
> If the deltaQuery contains variable like `last_index_time', the cached values never used increases.
> Similarly, I guess that the cache increases when fetching each modified row with primary key.
> I think these queries should not be cached. 
> I came up with two solutions:
>  1) Not to cache queries to get modified rows.
>  2) Make VariableResolverImpl#TEMPLATE_STRING non-static. Or clear cache on finishing delta-import.
> I think that #1 is better for performance than #2, but #2 is easier to solve the problem.
> I made a patch in #2 way, and then tested two solr applications with `-XX:+PrintClassHistgram' option.
> The result after importing several million rows from a MySQL database is as follows:
>  * original solr-1.3:
>  num     #instances         #bytes  class name
> ----------------------------------------------
> ...
>   6:       2983024      119320960  org.apache.solr.handler.dataimport.TemplateString
> ...
>  * patched solr-1.3:
>  num     #instances         #bytes  class name
> ----------------------------------------------
> ...
>  748:             3            120  org.apache.solr.handler.dataimport.TemplateString
> ...
> Though it is version 1.3 that I tested, perhaps current nightly version has same problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.