You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Enis Soztutar (JIRA)" <ji...@apache.org> on 2007/03/07 11:09:24 UTC
[jira] Created: (NUTCH-455) dedup on tokenized fields is faulty
dedup on tokenized fields is faulty
-----------------------------------
Key: NUTCH-455
URL: https://issues.apache.org/jira/browse/NUTCH-455
Project: Nutch
Issue Type: Bug
Components: searcher
Affects Versions: 0.9.0
Reporter: Enis Soztutar
Fix For: 0.9.0
(From LUCENE-252)
nutch uses several index servers, and the search results from these servers are merged using a dedup field for for deleting duplicates. The values from this field is cached by Lucene's FieldCachImpl. The default is the site field, which is indexed and tokenized. However for a Tokenized Field (for example "url" in nutch), FieldCacheImpl returns an array of Terms rather that array of field values, so dedup'ing becomes faulty. Current FieldCache implementation does not respect tokenized fields , and as described above caches only terms.
So in the situation that we are searching using "url" as the dedup field, when a Hit is constructed in IndexSearcher, the dedupValue becomes a token of the url (such as "www" or "com") rather that the whole url. This prevents using tokenized fields in the dedup field.
I have written a patch for lucene and attached it in http://issues.apache.org/jira/browse/LUCENE-252, this patch fixes the aforementioned issue about tokenized field caching. However building such a cache for about 1.5M documents takes 20+ secs. The code in IndexSearcher.translateHits() starts with
if (dedupField != null)
dedupValues = FieldCache.DEFAULT.getStrings(reader, dedupField);
and for the first call of search in IndexSearcher, cache is built.
Long story short, i have written a patch against IndexSearcher, which in constructor warms-up the caches of wanted fields(configurable). I think we should vote for LUCENE-252, and then commit the above patch with the last version of lucene.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-455) dedup on tokenized fields is faulty
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann updated NUTCH-455:
------------------------------------
Fix Version/s: (was: 1.1)
- pushing this out per http://bit.ly/c7tBv9
> dedup on tokenized fields is faulty
> -----------------------------------
>
> Key: NUTCH-455
> URL: https://issues.apache.org/jira/browse/NUTCH-455
> Project: Nutch
> Issue Type: Bug
> Components: searcher
> Affects Versions: 0.9.0
> Reporter: Enis Soztutar
> Attachments: IndexSearcherCacheWarm.patch
>
>
> (From LUCENE-252)
> nutch uses several index servers, and the search results from these servers are merged using a dedup field for for deleting duplicates. The values from this field is cached by Lucene's FieldCachImpl. The default is the site field, which is indexed and tokenized. However for a Tokenized Field (for example "url" in nutch), FieldCacheImpl returns an array of Terms rather that array of field values, so dedup'ing becomes faulty. Current FieldCache implementation does not respect tokenized fields , and as described above caches only terms.
> So in the situation that we are searching using "url" as the dedup field, when a Hit is constructed in IndexSearcher, the dedupValue becomes a token of the url (such as "www" or "com") rather that the whole url. This prevents using tokenized fields in the dedup field.
> I have written a patch for lucene and attached it in http://issues.apache.org/jira/browse/LUCENE-252, this patch fixes the aforementioned issue about tokenized field caching. However building such a cache for about 1.5M documents takes 20+ secs. The code in IndexSearcher.translateHits() starts with
> if (dedupField != null)
> dedupValues = FieldCache.DEFAULT.getStrings(reader, dedupField);
> and for the first call of search in IndexSearcher, cache is built.
> Long story short, i have written a patch against IndexSearcher, which in constructor warms-up the caches of wanted fields(configurable). I think we should vote for LUCENE-252, and then commit the above patch with the last version of lucene.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-455) dedup on tokenized fields is faulty
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Enis Soztutar updated NUTCH-455:
--------------------------------
Attachment: IndexSearcherCacheWarm.patch
the patch to the IndexSearcher is attached
> dedup on tokenized fields is faulty
> -----------------------------------
>
> Key: NUTCH-455
> URL: https://issues.apache.org/jira/browse/NUTCH-455
> Project: Nutch
> Issue Type: Bug
> Components: searcher
> Affects Versions: 0.9.0
> Reporter: Enis Soztutar
> Fix For: 0.9.0
>
> Attachments: IndexSearcherCacheWarm.patch
>
>
> (From LUCENE-252)
> nutch uses several index servers, and the search results from these servers are merged using a dedup field for for deleting duplicates. The values from this field is cached by Lucene's FieldCachImpl. The default is the site field, which is indexed and tokenized. However for a Tokenized Field (for example "url" in nutch), FieldCacheImpl returns an array of Terms rather that array of field values, so dedup'ing becomes faulty. Current FieldCache implementation does not respect tokenized fields , and as described above caches only terms.
> So in the situation that we are searching using "url" as the dedup field, when a Hit is constructed in IndexSearcher, the dedupValue becomes a token of the url (such as "www" or "com") rather that the whole url. This prevents using tokenized fields in the dedup field.
> I have written a patch for lucene and attached it in http://issues.apache.org/jira/browse/LUCENE-252, this patch fixes the aforementioned issue about tokenized field caching. However building such a cache for about 1.5M documents takes 20+ secs. The code in IndexSearcher.translateHits() starts with
> if (dedupField != null)
> dedupValues = FieldCache.DEFAULT.getStrings(reader, dedupField);
> and for the first call of search in IndexSearcher, cache is built.
> Long story short, i have written a patch against IndexSearcher, which in constructor warms-up the caches of wanted fields(configurable). I think we should vote for LUCENE-252, and then commit the above patch with the last version of lucene.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-455) dedup on tokenized fields is faulty
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12479262 ]
Enis Soztutar commented on NUTCH-455:
-------------------------------------
(from LUCENE-252)
In nutch we have 3 options : 1st is to disallow deleting duplicates on tokenized fields(due to FieldCache), 2nd is to index the tokenized field twice(once tokenized, and once untokenized), 3rd use LUCENE-252 and the above patch and warm the cache initially in the index servers.
I am in favor of the 3rd option.
I think first resolving LUCENE-252, and then proceeding with NUTCH-255 is more sensible.
> dedup on tokenized fields is faulty
> -----------------------------------
>
> Key: NUTCH-455
> URL: https://issues.apache.org/jira/browse/NUTCH-455
> Project: Nutch
> Issue Type: Bug
> Components: searcher
> Affects Versions: 0.9.0
> Reporter: Enis Soztutar
> Fix For: 0.9.0
>
> Attachments: IndexSearcherCacheWarm.patch
>
>
> (From LUCENE-252)
> nutch uses several index servers, and the search results from these servers are merged using a dedup field for for deleting duplicates. The values from this field is cached by Lucene's FieldCachImpl. The default is the site field, which is indexed and tokenized. However for a Tokenized Field (for example "url" in nutch), FieldCacheImpl returns an array of Terms rather that array of field values, so dedup'ing becomes faulty. Current FieldCache implementation does not respect tokenized fields , and as described above caches only terms.
> So in the situation that we are searching using "url" as the dedup field, when a Hit is constructed in IndexSearcher, the dedupValue becomes a token of the url (such as "www" or "com") rather that the whole url. This prevents using tokenized fields in the dedup field.
> I have written a patch for lucene and attached it in http://issues.apache.org/jira/browse/LUCENE-252, this patch fixes the aforementioned issue about tokenized field caching. However building such a cache for about 1.5M documents takes 20+ secs. The code in IndexSearcher.translateHits() starts with
> if (dedupField != null)
> dedupValues = FieldCache.DEFAULT.getStrings(reader, dedupField);
> and for the first call of search in IndexSearcher, cache is built.
> Long story short, i have written a patch against IndexSearcher, which in constructor warms-up the caches of wanted fields(configurable). I think we should vote for LUCENE-252, and then commit the above patch with the last version of lucene.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-455) dedup on tokenized fields is faulty
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12478854 ]
Doug Cutting commented on NUTCH-455:
------------------------------------
Alternately, we could define it as an error to attempt to dedup by a tokenized field. That's the (undocumented) expectation of FieldCache. Using documents to populate a FieldCache for tokenized fields is very slow. It's better to add an untokenized version and use that, no? If you agree, then the more appropriate fix is to document the restriction and try to check for it at runtime.
> dedup on tokenized fields is faulty
> -----------------------------------
>
> Key: NUTCH-455
> URL: https://issues.apache.org/jira/browse/NUTCH-455
> Project: Nutch
> Issue Type: Bug
> Components: searcher
> Affects Versions: 0.9.0
> Reporter: Enis Soztutar
> Fix For: 0.9.0
>
> Attachments: IndexSearcherCacheWarm.patch
>
>
> (From LUCENE-252)
> nutch uses several index servers, and the search results from these servers are merged using a dedup field for for deleting duplicates. The values from this field is cached by Lucene's FieldCachImpl. The default is the site field, which is indexed and tokenized. However for a Tokenized Field (for example "url" in nutch), FieldCacheImpl returns an array of Terms rather that array of field values, so dedup'ing becomes faulty. Current FieldCache implementation does not respect tokenized fields , and as described above caches only terms.
> So in the situation that we are searching using "url" as the dedup field, when a Hit is constructed in IndexSearcher, the dedupValue becomes a token of the url (such as "www" or "com") rather that the whole url. This prevents using tokenized fields in the dedup field.
> I have written a patch for lucene and attached it in http://issues.apache.org/jira/browse/LUCENE-252, this patch fixes the aforementioned issue about tokenized field caching. However building such a cache for about 1.5M documents takes 20+ secs. The code in IndexSearcher.translateHits() starts with
> if (dedupField != null)
> dedupValues = FieldCache.DEFAULT.getStrings(reader, dedupField);
> and for the first call of search in IndexSearcher, cache is built.
> Long story short, i have written a patch against IndexSearcher, which in constructor warms-up the caches of wanted fields(configurable). I think we should vote for LUCENE-252, and then commit the above patch with the last version of lucene.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-455) dedup on tokenized fields is faulty
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated NUTCH-455:
------------------------------------
Fix Version/s: (was: 1.0.0)
1.1
> dedup on tokenized fields is faulty
> -----------------------------------
>
> Key: NUTCH-455
> URL: https://issues.apache.org/jira/browse/NUTCH-455
> Project: Nutch
> Issue Type: Bug
> Components: searcher
> Affects Versions: 0.9.0
> Reporter: Enis Soztutar
> Fix For: 1.1
>
> Attachments: IndexSearcherCacheWarm.patch
>
>
> (From LUCENE-252)
> nutch uses several index servers, and the search results from these servers are merged using a dedup field for for deleting duplicates. The values from this field is cached by Lucene's FieldCachImpl. The default is the site field, which is indexed and tokenized. However for a Tokenized Field (for example "url" in nutch), FieldCacheImpl returns an array of Terms rather that array of field values, so dedup'ing becomes faulty. Current FieldCache implementation does not respect tokenized fields , and as described above caches only terms.
> So in the situation that we are searching using "url" as the dedup field, when a Hit is constructed in IndexSearcher, the dedupValue becomes a token of the url (such as "www" or "com") rather that the whole url. This prevents using tokenized fields in the dedup field.
> I have written a patch for lucene and attached it in http://issues.apache.org/jira/browse/LUCENE-252, this patch fixes the aforementioned issue about tokenized field caching. However building such a cache for about 1.5M documents takes 20+ secs. The code in IndexSearcher.translateHits() starts with
> if (dedupField != null)
> dedupValues = FieldCache.DEFAULT.getStrings(reader, dedupField);
> and for the first call of search in IndexSearcher, cache is built.
> Long story short, i have written a patch against IndexSearcher, which in constructor warms-up the caches of wanted fields(configurable). I think we should vote for LUCENE-252, and then commit the above patch with the last version of lucene.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-455) dedup on tokenized fields is faulty
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671134#action_12671134 ]
Andrzej Bialecki commented on NUTCH-455:
-----------------------------------------
Since LUCENE-252 is still unresolved, and it's not clear which of the proposed solutions should be selected, I'm postponing this issue.
> dedup on tokenized fields is faulty
> -----------------------------------
>
> Key: NUTCH-455
> URL: https://issues.apache.org/jira/browse/NUTCH-455
> Project: Nutch
> Issue Type: Bug
> Components: searcher
> Affects Versions: 0.9.0
> Reporter: Enis Soztutar
> Fix For: 1.1
>
> Attachments: IndexSearcherCacheWarm.patch
>
>
> (From LUCENE-252)
> nutch uses several index servers, and the search results from these servers are merged using a dedup field for for deleting duplicates. The values from this field is cached by Lucene's FieldCachImpl. The default is the site field, which is indexed and tokenized. However for a Tokenized Field (for example "url" in nutch), FieldCacheImpl returns an array of Terms rather that array of field values, so dedup'ing becomes faulty. Current FieldCache implementation does not respect tokenized fields , and as described above caches only terms.
> So in the situation that we are searching using "url" as the dedup field, when a Hit is constructed in IndexSearcher, the dedupValue becomes a token of the url (such as "www" or "com") rather that the whole url. This prevents using tokenized fields in the dedup field.
> I have written a patch for lucene and attached it in http://issues.apache.org/jira/browse/LUCENE-252, this patch fixes the aforementioned issue about tokenized field caching. However building such a cache for about 1.5M documents takes 20+ secs. The code in IndexSearcher.translateHits() starts with
> if (dedupField != null)
> dedupValues = FieldCache.DEFAULT.getStrings(reader, dedupField);
> and for the first call of search in IndexSearcher, cache is built.
> Long story short, i have written a patch against IndexSearcher, which in constructor warms-up the caches of wanted fields(configurable). I think we should vote for LUCENE-252, and then commit the above patch with the last version of lucene.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.