You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (JIRA)" <ji...@apache.org> on 2011/07/10 01:07:59 UTC
[jira] [Created] (NUTCH-1037) Deduplicate anchors before indexing
Deduplicate anchors before indexing
-----------------------------------
Key: NUTCH-1037
URL: https://issues.apache.org/jira/browse/NUTCH-1037
Project: Nutch
Issue Type: Improvement
Reporter: Markus Jelsma
Assignee: Markus Jelsma
Fix For: 1.4, 2.0
Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1037:
---------------------------------
Description:
Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
Use anchorIndexingFilter.deduplicate=true to deduplicate anchors case-insensitive.
was:
Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
Should this be implemented as a fix or as a new feature that needs to be configured?
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-1.4-2.patch, NUTCH-1037-1.4-3.patch, NUTCH-1037-2.0-1.patch, NUTCH-1037-2.0-2.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Use anchorIndexingFilter.deduplicate=true to deduplicate anchors case-insensitive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067668#comment-13067668 ]
Julien Nioche commented on NUTCH-1037:
--------------------------------------
+1. Maybe add to the description something like
{code}
This prevents possible hundreds or thousands of identical anchors for a given page to be indexed but will affect the search scoring.
{code}
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-1.4-2.patch, NUTCH-1037-1.4-3.patch, NUTCH-1037-2.0-1.patch, NUTCH-1037-2.0-2.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063375#comment-13063375 ]
Markus Jelsma commented on NUTCH-1037:
--------------------------------------
mmmmm, yes.
It would require changing existing indices for something that is not yet being fully supported out-of-the-box by Solr. A reason for not having used payloads as of yet. If i'm not mistaken it'll encode a byte for each term in the field, not for the whole field value so i think that would make things difficult, wouldn't it?
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-2.0-1.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067690#comment-13067690 ]
Markus Jelsma commented on NUTCH-1037:
--------------------------------------
Thanks. The comment has been modified.
Committed for 1.4 in rev. 1148305.
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-1.4-2.patch, NUTCH-1037-1.4-3.patch, NUTCH-1037-2.0-1.patch, NUTCH-1037-2.0-2.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma closed NUTCH-1037.
--------------------------------
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-1.4-2.patch, NUTCH-1037-1.4-3.patch, NUTCH-1037-2.0-1.patch, NUTCH-1037-2.0-2.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Use anchorIndexingFilter.deduplicate=true to deduplicate anchors case-insensitive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1037:
---------------------------------
Patch Info: [Patch Available]
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-2.0-1.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1037:
---------------------------------
Attachment: NUTCH-1037-2.0-2.patch
NUTCH-1037-1.4-2.patch
New patches for 1.4 and trunk. The feature is now optional (and disabled by default). Use anchorIndexingFilter.deduplicate to toggle.
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-1.4-2.patch, NUTCH-1037-2.0-1.patch, NUTCH-1037-2.0-2.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063363#comment-13063363 ]
Julien Nioche commented on NUTCH-1037:
--------------------------------------
Deduplicating the anchors would have an impact on the score on the search side with all the terms having a doc freq of 1. As a result an anchor appearing once for a doc would yield the same score as one appearing 100 times which is not right.
A less drastic option would be to pass the number of occurrences for a given anchor using the payloads format. This is partially supported in SOLR - see http://digitalpebble.blogspot.com/2010/08/using-payloads-with-dismaxqparser-in.html for a workaround. This way, we'd send less data to SOLR for indexing while retaining the information about the frequencies and being able to take that into account for the scoring
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-2.0-1.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1037:
---------------------------------
Attachment: NUTCH-1037-1.4-3.patch
Of course. Patch doesn't lowercase anchors and added config option.
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-1.4-2.patch, NUTCH-1037-1.4-3.patch, NUTCH-1037-2.0-1.patch, NUTCH-1037-2.0-2.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063383#comment-13063383 ]
Markus Jelsma commented on NUTCH-1037:
--------------------------------------
Shall i then make this feature optional and open a separate issue for encoding frequency using payloads?
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-2.0-1.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1037:
---------------------------------
Attachment: NUTCH-1037-1.4-1.patch
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (NUTCH-1037) Deduplicate anchors
before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063396#comment-13063396 ]
Markus Jelsma edited comment on NUTCH-1037 at 7/11/11 3:47 PM:
---------------------------------------------------------------
New patches for 1.4 and trunk. The feature is now optional (and disabled by default). Use anchorIndexingFilter.deduplicate to toggle.
edit: if this needs to be committed i'll demote the `not adding duplicate anchor`-warning to debug level.
was (Author: markus17):
New patches for 1.4 and trunk. The feature is now optional (and disabled by default). Use anchorIndexingFilter.deduplicate to toggle.
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-1.4-2.patch, NUTCH-1037-2.0-1.patch, NUTCH-1037-2.0-2.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062780#comment-13062780 ]
Markus Jelsma commented on NUTCH-1037:
--------------------------------------
I'd choose a fix instead a new config option. I've done more checks and see no point in having many hundreds (really!) exact or case-similar values for the anchors field. It slows down indexing dramatically (when working in the millions) and increases index size.
Comments?
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064612#comment-13064612 ]
Markus Jelsma commented on NUTCH-1037:
--------------------------------------
* indentation where i add the lowercased key to the map you mean? Misses two spaces indeed;
* will add config to default + motivation
* what do you mean by `lowercase the when duduping (but not sending)?
* agreed on holding the commit
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-1.4-2.patch, NUTCH-1037-2.0-1.patch, NUTCH-1037-2.0-2.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064618#comment-13064618 ]
Julien Nioche commented on NUTCH-1037:
--------------------------------------
* indentation : not that bad indeed - must be my eyes getting tired :-)
* lowercase the when deduplicating (but not sending)=> you lowercase the anchors before checking whether have already been found and if not send them as they are to SOLR (which is the right thing to do as it's up to SOLR to take care of the analysis)
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-1.4-2.patch, NUTCH-1037-2.0-1.patch, NUTCH-1037-2.0-2.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063380#comment-13063380 ]
Julien Nioche commented on NUTCH-1037:
--------------------------------------
IIRC it stores a byte array for each term, there are utilities to simplify the coding to the byte array.
We could distribute the occurrences for the whole anchor to each individual term it contains. Have done that for an ex-client, worked a treat.
Only trouble as you pointed out is that the search part of it is not supported by default is SOLR (yet), so this is not an option for now but could be used later instead of the deduplication you suggested which would have too much impact on the scores.
Of course if the deduplication is optional then that's fine and people can choose between having smaller docs vs more relevant search on the anchors
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-2.0-1.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067661#comment-13067661 ]
Markus Jelsma commented on NUTCH-1037:
--------------------------------------
Any comments guys? Julien's notes have been implemented.
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-1.4-2.patch, NUTCH-1037-1.4-3.patch, NUTCH-1037-2.0-1.patch, NUTCH-1037-2.0-2.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063337#comment-13063337 ]
Markus Jelsma commented on NUTCH-1037:
--------------------------------------
Wrong patches, in these i somehow omitted the negation operator in the condition. Add it. This fix is then confirmed to work in large environments.
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-2.0-1.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064587#comment-13064587 ]
Markus Jelsma commented on NUTCH-1037:
--------------------------------------
If there are no objections to having this feature optional, i'll see if i can commit this today.
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-1.4-2.patch, NUTCH-1037-2.0-1.patch, NUTCH-1037-2.0-2.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064596#comment-13064596 ]
Julien Nioche commented on NUTCH-1037:
--------------------------------------
Looks OK apart from the indentation + need to add to nutch-default and explain the motivations for this + that you lowercase the when deduplicating (but not sending)
PS: could you wait a bit more than a couple of days before committing? I think it's a good practice to have at least a +1 from another committer just to double check
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-1.4-2.patch, NUTCH-1037-2.0-1.patch, NUTCH-1037-2.0-2.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma resolved NUTCH-1037.
----------------------------------
Resolution: Fixed
Committed for trunk in rev. 1148308.
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-1.4-2.patch, NUTCH-1037-1.4-3.patch, NUTCH-1037-2.0-1.patch, NUTCH-1037-2.0-2.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068149#comment-13068149 ]
Hudson commented on NUTCH-1037:
-------------------------------
Integrated in Nutch-trunk #1551 (See [https://builds.apache.org/job/Nutch-trunk/1551/])
NUTCH-1037 Option to deduplicate anchors prior to indexing
markus : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1148308
Files :
* /nutch/trunk/conf/nutch-default.xml
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/src/plugin/index-anchor/src/java/org/apache/nutch/indexer/anchor/AnchorIndexingFilter.java
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-1.4-2.patch, NUTCH-1037-1.4-3.patch, NUTCH-1037-2.0-1.patch, NUTCH-1037-2.0-2.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Use anchorIndexingFilter.deduplicate=true to deduplicate anchors case-insensitive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (NUTCH-1037) Deduplicate anchors
before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063375#comment-13063375 ]
Markus Jelsma edited comment on NUTCH-1037 at 7/11/11 2:53 PM:
---------------------------------------------------------------
mmmmm, yes.
It would require changing existing indices for something that is not yet being fully supported out-of-the-box by Solr. A reason for not having used payloads as of yet. If i'm not mistaken it'll encode a byte for each term in the field, not for the group of terms that make up an anchor, so i think that would make things difficult, wouldn't it?
was (Author: markus17):
mmmmm, yes.
It would require changing existing indices for something that is not yet being fully supported out-of-the-box by Solr. A reason for not having used payloads as of yet. If i'm not mistaken it'll encode a byte for each term in the field, not for the whole field value so i think that would make things difficult, wouldn't it?
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-2.0-1.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1037) Deduplicate anchors before indexing
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1037:
---------------------------------
Attachment: NUTCH-1037-2.0-1.patch
Patch for 2.0.
> Deduplicate anchors before indexing
> -----------------------------------
>
> Key: NUTCH-1037
> URL: https://issues.apache.org/jira/browse/NUTCH-1037
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1037-1.4-1.patch, NUTCH-1037-2.0-1.patch
>
>
> Anchors are not deduplicated before indexing. This can result in a very high number of similar and identical anchors being indexed. Before indexing, anchors must be deduplicated at least on case.
> Should this be implemented as a fix or as a new feature that needs to be configured?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira