You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "congliu (JIRA)" <ji...@apache.org> on 2010/10/23 11:36:19 UTC

[jira] Created: (NUTCH-925) plugins stored in weakhashmap lead memory leak

plugins stored in weakhashmap lead memory leak
----------------------------------------------

                 Key: NUTCH-925
                 URL: https://issues.apache.org/jira/browse/NUTCH-925
             Project: Nutch
          Issue Type: Bug
    Affects Versions: 1.2
            Reporter: congliu


I suffer serious memory leak using Nutch 1.2 though a very deep crawl. I get the error like this:

Exception in thread "Thread-113544" java.lang.OutOfMemoryError: PermGen space
	at java.lang.Throwable.getStackTraceElement(Native Method)
	at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
	at java.lang.Throwable.printStackTrace(Throwable.java:510)
	at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:76)
	at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:407)
	at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:305)
	at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
	at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
	at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
	at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
	at org.apache.log4j.Category.callAppenders(Category.java:206)
	at org.apache.log4j.Category.forcedLog(Category.java:391)
	at org.apache.log4j.Category.log(Category.java:856)
	at org.slf4j.impl.Log4jLoggerAdapter.log(Log4jLoggerAdapter.java:509)
	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.warn(SLF4JLocationAwareLog.java:173)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:256)
Exception in thread "main" java.io.IOException: Job failed!
	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
	at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1107)
	at org.apache.nutch.crawl.Crawl.main(Crawl.java:133)


I guess Plugin repository cache lead to memory leak.

As u know plugins is stored in weakhashmap <conf, plugins>, and new class classload
create when u need plugins.
Usually,WeakHashMap object can been gc, but class and classload is stored in Perm NOT stack and gc can't perform in Perm, SO (java.lang.OutOfMemoryError: PermGen space) occured..., is any nutch-issues have concerned this promble? or there is any solution? 

nutch-356 may help?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] [Closed] (NUTCH-925) plugins stored in weakhashmap lead memory leak

Posted by "Markus Jelsma (Closed) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/NUTCH-925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Markus Jelsma closed NUTCH-925.
-------------------------------


Bulk close of resolved issues of 1.4. bulkclose-1.4-20111220
                
> plugins stored in weakhashmap lead memory leak
> ----------------------------------------------
>
>                 Key: NUTCH-925
>                 URL: https://issues.apache.org/jira/browse/NUTCH-925
>             Project: Nutch
>          Issue Type: Bug
>    Affects Versions: 1.2
>            Reporter: congliu
>             Fix For: 1.4
>
>
> I suffer serious memory leak using Nutch 1.2 though a very deep crawl. I get the error like this:
> Exception in thread "Thread-113544" java.lang.OutOfMemoryError: PermGen space
> 	at java.lang.Throwable.getStackTraceElement(Native Method)
> 	at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
> 	at java.lang.Throwable.printStackTrace(Throwable.java:510)
> 	at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:76)
> 	at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:407)
> 	at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:305)
> 	at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> 	at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> 	at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> 	at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> 	at org.apache.log4j.Category.callAppenders(Category.java:206)
> 	at org.apache.log4j.Category.forcedLog(Category.java:391)
> 	at org.apache.log4j.Category.log(Category.java:856)
> 	at org.slf4j.impl.Log4jLoggerAdapter.log(Log4jLoggerAdapter.java:509)
> 	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.warn(SLF4JLocationAwareLog.java:173)
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:256)
> Exception in thread "main" java.io.IOException: Job failed!
> 	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> 	at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1107)
> 	at org.apache.nutch.crawl.Crawl.main(Crawl.java:133)
> I guess Plugin repository cache lead to memory leak.
> As u know plugins is stored in weakhashmap <conf, plugins>, and new class classload
> create when u need plugins.
> Usually,WeakHashMap object can been gc, but class and classload is stored in Perm NOT stack and gc can't perform in Perm, SO (java.lang.OutOfMemoryError: PermGen space) occured..., is any nutch-issues have concerned this promble? or there is any solution? 
> nutch-356 may help?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (NUTCH-925) plugins stored in weakhashmap lead memory leak

Posted by "congliu (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/NUTCH-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173178#comment-13173178 ] 

congliu commented on NUTCH-925:
-------------------------------

Your guess is right, there already are solution to this problem. 






                
> plugins stored in weakhashmap lead memory leak
> ----------------------------------------------
>
>                 Key: NUTCH-925
>                 URL: https://issues.apache.org/jira/browse/NUTCH-925
>             Project: Nutch
>          Issue Type: Bug
>    Affects Versions: 1.2
>            Reporter: congliu
>             Fix For: 1.4
>
>
> I suffer serious memory leak using Nutch 1.2 though a very deep crawl. I get the error like this:
> Exception in thread "Thread-113544" java.lang.OutOfMemoryError: PermGen space
> 	at java.lang.Throwable.getStackTraceElement(Native Method)
> 	at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
> 	at java.lang.Throwable.printStackTrace(Throwable.java:510)
> 	at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:76)
> 	at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:407)
> 	at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:305)
> 	at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> 	at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> 	at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> 	at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> 	at org.apache.log4j.Category.callAppenders(Category.java:206)
> 	at org.apache.log4j.Category.forcedLog(Category.java:391)
> 	at org.apache.log4j.Category.log(Category.java:856)
> 	at org.slf4j.impl.Log4jLoggerAdapter.log(Log4jLoggerAdapter.java:509)
> 	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.warn(SLF4JLocationAwareLog.java:173)
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:256)
> Exception in thread "main" java.io.IOException: Job failed!
> 	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> 	at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1107)
> 	at org.apache.nutch.crawl.Crawl.main(Crawl.java:133)
> I guess Plugin repository cache lead to memory leak.
> As u know plugins is stored in weakhashmap <conf, plugins>, and new class classload
> create when u need plugins.
> Usually,WeakHashMap object can been gc, but class and classload is stored in Perm NOT stack and gc can't perform in Perm, SO (java.lang.OutOfMemoryError: PermGen space) occured..., is any nutch-issues have concerned this promble? or there is any solution? 
> nutch-356 may help?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (NUTCH-925) plugins stored in weakhashmap lead memory leak

Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/NUTCH-925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Markus Jelsma updated NUTCH-925:
--------------------------------

    Fix Version/s: 1.4

This has been fixed for 2.0 in NUTCH-844 but not in 1.x.

> plugins stored in weakhashmap lead memory leak
> ----------------------------------------------
>
>                 Key: NUTCH-925
>                 URL: https://issues.apache.org/jira/browse/NUTCH-925
>             Project: Nutch
>          Issue Type: Bug
>    Affects Versions: 1.2
>            Reporter: congliu
>             Fix For: 1.4
>
>
> I suffer serious memory leak using Nutch 1.2 though a very deep crawl. I get the error like this:
> Exception in thread "Thread-113544" java.lang.OutOfMemoryError: PermGen space
> 	at java.lang.Throwable.getStackTraceElement(Native Method)
> 	at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
> 	at java.lang.Throwable.printStackTrace(Throwable.java:510)
> 	at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:76)
> 	at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:407)
> 	at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:305)
> 	at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> 	at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> 	at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> 	at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> 	at org.apache.log4j.Category.callAppenders(Category.java:206)
> 	at org.apache.log4j.Category.forcedLog(Category.java:391)
> 	at org.apache.log4j.Category.log(Category.java:856)
> 	at org.slf4j.impl.Log4jLoggerAdapter.log(Log4jLoggerAdapter.java:509)
> 	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.warn(SLF4JLocationAwareLog.java:173)
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:256)
> Exception in thread "main" java.io.IOException: Job failed!
> 	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> 	at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1107)
> 	at org.apache.nutch.crawl.Crawl.main(Crawl.java:133)
> I guess Plugin repository cache lead to memory leak.
> As u know plugins is stored in weakhashmap <conf, plugins>, and new class classload
> create when u need plugins.
> Usually,WeakHashMap object can been gc, but class and classload is stored in Perm NOT stack and gc can't perform in Perm, SO (java.lang.OutOfMemoryError: PermGen space) occured..., is any nutch-issues have concerned this promble? or there is any solution? 
> nutch-356 may help?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (NUTCH-925) plugins stored in weakhashmap lead memory leak

Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/NUTCH-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061291#comment-13061291 ] 

Julien Nioche commented on NUTCH-925:
-------------------------------------

This should have been part of the batch of issues which I backported from 2.0 to 1.x. Can't see it in the list of modifs in 1.3 though so it is possible that it slipped through the net.

> plugins stored in weakhashmap lead memory leak
> ----------------------------------------------
>
>                 Key: NUTCH-925
>                 URL: https://issues.apache.org/jira/browse/NUTCH-925
>             Project: Nutch
>          Issue Type: Bug
>    Affects Versions: 1.2
>            Reporter: congliu
>             Fix For: 1.4
>
>
> I suffer serious memory leak using Nutch 1.2 though a very deep crawl. I get the error like this:
> Exception in thread "Thread-113544" java.lang.OutOfMemoryError: PermGen space
> 	at java.lang.Throwable.getStackTraceElement(Native Method)
> 	at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
> 	at java.lang.Throwable.printStackTrace(Throwable.java:510)
> 	at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:76)
> 	at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:407)
> 	at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:305)
> 	at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> 	at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> 	at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> 	at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> 	at org.apache.log4j.Category.callAppenders(Category.java:206)
> 	at org.apache.log4j.Category.forcedLog(Category.java:391)
> 	at org.apache.log4j.Category.log(Category.java:856)
> 	at org.slf4j.impl.Log4jLoggerAdapter.log(Log4jLoggerAdapter.java:509)
> 	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.warn(SLF4JLocationAwareLog.java:173)
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:256)
> Exception in thread "main" java.io.IOException: Job failed!
> 	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> 	at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1107)
> 	at org.apache.nutch.crawl.Crawl.main(Crawl.java:133)
> I guess Plugin repository cache lead to memory leak.
> As u know plugins is stored in weakhashmap <conf, plugins>, and new class classload
> create when u need plugins.
> Usually,WeakHashMap object can been gc, but class and classload is stored in Perm NOT stack and gc can't perform in Perm, SO (java.lang.OutOfMemoryError: PermGen space) occured..., is any nutch-issues have concerned this promble? or there is any solution? 
> nutch-356 may help?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (NUTCH-925) plugins stored in weakhashmap lead memory leak

Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/NUTCH-925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Markus Jelsma resolved NUTCH-925.
---------------------------------

    Resolution: Duplicate

I've checked the PluginReposiry diff of NUTCH-844 and compared with 1.4. It's in!

> plugins stored in weakhashmap lead memory leak
> ----------------------------------------------
>
>                 Key: NUTCH-925
>                 URL: https://issues.apache.org/jira/browse/NUTCH-925
>             Project: Nutch
>          Issue Type: Bug
>    Affects Versions: 1.2
>            Reporter: congliu
>             Fix For: 1.4
>
>
> I suffer serious memory leak using Nutch 1.2 though a very deep crawl. I get the error like this:
> Exception in thread "Thread-113544" java.lang.OutOfMemoryError: PermGen space
> 	at java.lang.Throwable.getStackTraceElement(Native Method)
> 	at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
> 	at java.lang.Throwable.printStackTrace(Throwable.java:510)
> 	at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:76)
> 	at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:407)
> 	at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:305)
> 	at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
> 	at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
> 	at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> 	at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> 	at org.apache.log4j.Category.callAppenders(Category.java:206)
> 	at org.apache.log4j.Category.forcedLog(Category.java:391)
> 	at org.apache.log4j.Category.log(Category.java:856)
> 	at org.slf4j.impl.Log4jLoggerAdapter.log(Log4jLoggerAdapter.java:509)
> 	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.warn(SLF4JLocationAwareLog.java:173)
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:256)
> Exception in thread "main" java.io.IOException: Job failed!
> 	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> 	at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1107)
> 	at org.apache.nutch.crawl.Crawl.main(Crawl.java:133)
> I guess Plugin repository cache lead to memory leak.
> As u know plugins is stored in weakhashmap <conf, plugins>, and new class classload
> create when u need plugins.
> Usually,WeakHashMap object can been gc, but class and classload is stored in Perm NOT stack and gc can't perform in Perm, SO (java.lang.OutOfMemoryError: PermGen space) occured..., is any nutch-issues have concerned this promble? or there is any solution? 
> nutch-356 may help?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira