You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (Created) (JIRA)" <ji...@apache.org> on 2011/11/02 18:43:32 UTC
[jira] [Created] (NUTCH-1194) CrawlDB lock should be released
earlier
CrawlDB lock should be released earlier
---------------------------------------
Key: NUTCH-1194
URL: https://issues.apache.org/jira/browse/NUTCH-1194
Project: Nutch
Issue Type: Improvement
Components: generator
Reporter: Markus Jelsma
Assignee: Markus Jelsma
Priority: Minor
Fix For: 1.5
Lock on the CrawlDB is released when everything is finished. But when generating many segments, the lock remains in place while it's not neccessary anymore. If GENERATE_UPDATE_DB is false we can release the lock immediately after the selector has finished.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1194) CrawlDB lock should be released
earlier
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143993#comment-13143993 ]
Markus Jelsma commented on NUTCH-1194:
--------------------------------------
The comment above mine was removed by the user who was writing about "locking should be done in setup/cleanup task.". This CrawlDB lock spans multiple mapred jobs which makes the difference.
> CrawlDB lock should be released earlier
> ---------------------------------------
>
> Key: NUTCH-1194
> URL: https://issues.apache.org/jira/browse/NUTCH-1194
> Project: Nutch
> Issue Type: Improvement
> Components: generator
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.5
>
>
> Lock on the CrawlDB is released when everything is finished. But when generating many segments, the lock remains in place while it's not neccessary anymore. If GENERATE_UPDATE_DB is false we can release the lock immediately after the selector has finished.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1194) CrawlDB lock should be released
earlier
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142659#comment-13142659 ]
Markus Jelsma commented on NUTCH-1194:
--------------------------------------
I'm not sure what you mean by setup and cleanup tasks, are you perhaps referring to the open() and close() methods of individual mapred tasks?
> CrawlDB lock should be released earlier
> ---------------------------------------
>
> Key: NUTCH-1194
> URL: https://issues.apache.org/jira/browse/NUTCH-1194
> Project: Nutch
> Issue Type: Improvement
> Components: generator
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.5
>
>
> Lock on the CrawlDB is released when everything is finished. But when generating many segments, the lock remains in place while it's not neccessary anymore. If GENERATE_UPDATE_DB is false we can release the lock immediately after the selector has finished.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1194) CrawlDB lock should be released
earlier
Posted by "Radim Kolar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142544#comment-13142544 ]
Radim Kolar commented on NUTCH-1194:
------------------------------------
locking should be done in setup/cleanup task. Currently if you kill process submitting generate job to hadoop then crawl database will stay locked. It needs to be reworked: instead of running jobs one by one, submit them all at once and make them depends on each other. After jobs are placed in hadoop queue you can kill client without causing any bad effects.
> CrawlDB lock should be released earlier
> ---------------------------------------
>
> Key: NUTCH-1194
> URL: https://issues.apache.org/jira/browse/NUTCH-1194
> Project: Nutch
> Issue Type: Improvement
> Components: generator
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.5
>
>
> Lock on the CrawlDB is released when everything is finished. But when generating many segments, the lock remains in place while it's not neccessary anymore. If GENERATE_UPDATE_DB is false we can release the lock immediately after the selector has finished.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1194) CrawlDB lock should be released
earlier
Posted by "Markus Jelsma (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1194:
---------------------------------
Fix Version/s: (was: 1.5)
1.6
20120304-push-1.6
> CrawlDB lock should be released earlier
> ---------------------------------------
>
> Key: NUTCH-1194
> URL: https://issues.apache.org/jira/browse/NUTCH-1194
> Project: Nutch
> Issue Type: Improvement
> Components: generator
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
>
> Lock on the CrawlDB is released when everything is finished. But when generating many segments, the lock remains in place while it's not neccessary anymore. If GENERATE_UPDATE_DB is false we can release the lock immediately after the selector has finished.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1194) CrawlDB lock should be released
earlier
Posted by "Radim Kolar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radim Kolar updated NUTCH-1194:
-------------------------------
Comment: was deleted
(was: locking should be done in setup/cleanup task. Currently if you kill process submitting generate job to hadoop then crawl database will stay locked. It needs to be reworked: instead of running jobs one by one, submit them all at once and make them depends on each other. After jobs are placed in hadoop queue you can kill client without causing any bad effects.)
> CrawlDB lock should be released earlier
> ---------------------------------------
>
> Key: NUTCH-1194
> URL: https://issues.apache.org/jira/browse/NUTCH-1194
> Project: Nutch
> Issue Type: Improvement
> Components: generator
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.5
>
>
> Lock on the CrawlDB is released when everything is finished. But when generating many segments, the lock remains in place while it's not neccessary anymore. If GENERATE_UPDATE_DB is false we can release the lock immediately after the selector has finished.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira