You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (Created) (JIRA)" <ji...@apache.org> on 2011/10/31 10:00:34 UTC
[jira] [Created] (NUTCH-1185) Decrease solr.commit.size
Decrease solr.commit.size
-------------------------
Key: NUTCH-1185
URL: https://issues.apache.org/jira/browse/NUTCH-1185
Project: Nutch
Issue Type: Task
Components: indexer
Affects Versions: 1.3
Reporter: Markus Jelsma
Assignee: Markus Jelsma
Priority: Trivial
Fix For: 1.5
Default document batch size should be decreased to prevent OOMEs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1185) Decrease solr.commit.size
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140058#comment-13140058 ]
Markus Jelsma commented on NUTCH-1185:
--------------------------------------
Would that be feasible? It's not easy to restore to a previous state and resume differently and we don't really know where it's thrown from. The Javadocs on Error states that these should not be caught. Also, a failing reducer will retry anyway for a number of times.
Do you think it's worth the effort?
> Decrease solr.commit.size
> -------------------------
>
> Key: NUTCH-1185
> URL: https://issues.apache.org/jira/browse/NUTCH-1185
> Project: Nutch
> Issue Type: Task
> Components: indexer
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Trivial
> Fix For: 1.5
>
>
> Default document batch size should be decreased to prevent OOMEs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1185) Decrease solr.commit.size
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148999#comment-13148999 ]
Hudson commented on NUTCH-1185:
-------------------------------
Integrated in Nutch-trunk #1660 (See [https://builds.apache.org/job/Nutch-trunk/1660/])
NUTCH-1185 Decrease solr.commit.size to 250
markus : http://svn.apache.org/viewvc/nutch/trunk/viewvc/?view=rev&root=&revision=1200833
Files :
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/conf/nutch-default.xml
> Decrease solr.commit.size
> -------------------------
>
> Key: NUTCH-1185
> URL: https://issues.apache.org/jira/browse/NUTCH-1185
> Project: Nutch
> Issue Type: Task
> Components: indexer
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Trivial
> Fix For: 1.5
>
>
> Default document batch size should be decreased to prevent OOMEs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (NUTCH-1185) Decrease solr.commit.size
Posted by "Markus Jelsma (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma resolved NUTCH-1185.
----------------------------------
Resolution: Fixed
Committed for nutchgora in rev. 1200834. and for trunk in rev. 1200833.
> Decrease solr.commit.size
> -------------------------
>
> Key: NUTCH-1185
> URL: https://issues.apache.org/jira/browse/NUTCH-1185
> Project: Nutch
> Issue Type: Task
> Components: indexer
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Trivial
> Fix For: 1.5
>
>
> Default document batch size should be decreased to prevent OOMEs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1185) Decrease solr.commit.size
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149029#comment-13149029 ]
Hudson commented on NUTCH-1185:
-------------------------------
Integrated in Nutch-trunk-ant #77 (See [https://builds.apache.org/job/Nutch-trunk-ant/77/])
NUTCH-1185 Decrease solr.commit.size to 250
markus : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1200833
Files :
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/conf/nutch-default.xml
> Decrease solr.commit.size
> -------------------------
>
> Key: NUTCH-1185
> URL: https://issues.apache.org/jira/browse/NUTCH-1185
> Project: Nutch
> Issue Type: Task
> Components: indexer
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Trivial
> Fix For: 1.5
>
>
> Default document batch size should be decreased to prevent OOMEs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1185) Decrease solr.commit.size
Posted by "Julien Nioche (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140047#comment-13140047 ]
Julien Nioche commented on NUTCH-1185:
--------------------------------------
Or we could catch the exceptions (OOME or else) and retry the docs in the buffer one by one which will also help identify when a specific document is causing problem
> Decrease solr.commit.size
> -------------------------
>
> Key: NUTCH-1185
> URL: https://issues.apache.org/jira/browse/NUTCH-1185
> Project: Nutch
> Issue Type: Task
> Components: indexer
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Trivial
> Fix For: 1.5
>
>
> Default document batch size should be decreased to prevent OOMEs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1185) Decrease solr.commit.size
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149037#comment-13149037 ]
Hudson commented on NUTCH-1185:
-------------------------------
Integrated in Nutch-nutchgora #65 (See [https://builds.apache.org/job/Nutch-nutchgora/65/])
NUTCH-1185 Decrease solr.commit.size to 250
markus : http://svn.apache.org/viewvc/nutch/branches/nutchgora/viewvc/?view=rev&root=&revision=1200834
Files :
* /nutch/branches/nutchgora/CHANGES.txt
* /nutch/branches/nutchgora/conf/nutch-default.xml
> Decrease solr.commit.size
> -------------------------
>
> Key: NUTCH-1185
> URL: https://issues.apache.org/jira/browse/NUTCH-1185
> Project: Nutch
> Issue Type: Task
> Components: indexer
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Trivial
> Fix For: 1.5
>
>
> Default document batch size should be decreased to prevent OOMEs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1185) Decrease solr.commit.size
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149038#comment-13149038 ]
Hudson commented on NUTCH-1185:
-------------------------------
Integrated in Nutch-nutchgora-ant #19 (See [https://builds.apache.org/job/Nutch-nutchgora-ant/19/])
NUTCH-1185 Decrease solr.commit.size to 250
markus : http://svn.apache.org/viewvc/nutch/branches/nutchgora/viewvc/?view=rev&root=&revision=1200834
Files :
* /nutch/branches/nutchgora/CHANGES.txt
* /nutch/branches/nutchgora/conf/nutch-default.xml
> Decrease solr.commit.size
> -------------------------
>
> Key: NUTCH-1185
> URL: https://issues.apache.org/jira/browse/NUTCH-1185
> Project: Nutch
> Issue Type: Task
> Components: indexer
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Trivial
> Fix For: 1.5
>
>
> Default document batch size should be decreased to prevent OOMEs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1185) Decrease solr.commit.size
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148443#comment-13148443 ]
Hudson commented on NUTCH-1185:
-------------------------------
Integrated in nutch-trunk-maven #19 (See [https://builds.apache.org/job/nutch-trunk-maven/19/])
NUTCH-1185 Decrease solr.commit.size to 250
markus : http://svn.apache.org/viewvc/nutch/trunk/viewvc/?view=rev&root=&revision=1200833
Files :
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/conf/nutch-default.xml
> Decrease solr.commit.size
> -------------------------
>
> Key: NUTCH-1185
> URL: https://issues.apache.org/jira/browse/NUTCH-1185
> Project: Nutch
> Issue Type: Task
> Components: indexer
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Trivial
> Fix For: 1.5
>
>
> Default document batch size should be decreased to prevent OOMEs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira