You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (Created) (JIRA)" <ji...@apache.org> on 2011/10/01 13:57:33 UTC
[jira] [Created] (NUTCH-1143) Omit anchor in webgraph's LinkDatum
Omit anchor in webgraph's LinkDatum
-----------------------------------
Key: NUTCH-1143
URL: https://issues.apache.org/jira/browse/NUTCH-1143
Project: Nutch
Issue Type: Improvement
Reporter: Markus Jelsma
Assignee: Markus Jelsma
Priority: Minor
Fix For: 1.5
Anchors are stored unchecked in the webgraph. it looks like for cosmetic reasons only. When dealing with hundreds of millions of records it takes up significant space and I/O time.
This issue should add an option to omit the anchor.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1143) Omit anchor in webgraph's LinkDatum
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119281#comment-13119281 ]
Markus Jelsma commented on NUTCH-1143:
--------------------------------------
It seems the anchor field was once used for indexing the best ranking anchor for a given URL but the indexing code is legacy. With the current version users must invert links and pass the linkdb and enable index-anchor to index anchors so having an anchor in LinkDatum is obsolete for now.
Instead of completely removing the anchor code we should make it optional, by doing that we can write indexing code later and pass the webgraph to the indexer instead of a linkdb.
I opt for defaulting the setting to false (i.e. do not store anchors) since they are unusable at the moment.
> Omit anchor in webgraph's LinkDatum
> -----------------------------------
>
> Key: NUTCH-1143
> URL: https://issues.apache.org/jira/browse/NUTCH-1143
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.5
>
>
> Anchors are stored unchecked in the webgraph. it looks like for cosmetic reasons only. When dealing with hundreds of millions of records it takes up significant space and I/O time.
> This issue should add an option to omit the anchor.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1143) Omit anchor in webgraph's LinkDatum
Posted by "Markus Jelsma (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1143:
---------------------------------
Fix Version/s: (was: 1.5)
1.6
20120304-push-1.6
> Omit anchor in webgraph's LinkDatum
> -----------------------------------
>
> Key: NUTCH-1143
> URL: https://issues.apache.org/jira/browse/NUTCH-1143
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
>
> Anchors are stored unchecked in the webgraph. it looks like for cosmetic reasons only. When dealing with hundreds of millions of records it takes up significant space and I/O time.
> This issue should add an option to omit the anchor.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira