You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Herman Kiefus <he...@angieslist.com> on 2011/08/22 15:25:35 UTC
Text Analysis and copyField
Is my thinking correct?
I have a field 'F1' of type 'T1' whose index time analysis employs the StopFilterFactory.
I also have a field 'F2' of type 'T2' whose index time analysis does NOT employ the StopFilterFactory.
There is a copyField directive source="F1" dest="F2"
F2 will not contain any stop words because they were filtered out as F1 was populated.
RE: Text Analysis and copyField
Posted by Herman Kiefus <he...@angieslist.com>.
It had crossed my mind but for now we have a 'DictionarySource' field whose type utilizes the KeepWordFilterFactory that uses a text file containing all correctly spelled words (thanks to scrabble), location/last/first names (courtesy of the US census bureau) and a few other adds (month/day) names. A file this large does not seem to have a material impact on indexing.
What we're seeing now (we also have a field 'TermsMisspelled' that utilizes the same text file with StopFilterFactory) is almost pure misspellings and some contractions (can't, won't, don't, etc.).
Thank you everyone for your help here, this is a truly fine community.
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Wednesday, August 24, 2011 1:00 PM
To: solr-user@lucene.apache.org
Subject: Re: Text Analysis and copyField
Have you considered having two dictionaries and using ajax to query them both and intermingling the results in your suggestions? It'd be some work, but I think it might accomplish what you want.
Best
Erick
On Tue, Aug 23, 2011 at 1:48 PM, Herman Kiefus <he...@angieslist.com> wrote:
> To close, I found this article from Hoss:
> http://lucene.472066.n3.nabble.com/CopyField-into-another-CopyField-td
> 3122408.html
>
> Since I cannot use one copyField directive to copy from another copyField's dest[ination], I cannot achieve what I desire: some terms that are subject to KeepWordFilterFactory and some that are not.
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Monday, August 22, 2011 1:16 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Text Analysis and copyField
>
> I suspect that the things going into TermsDictionary are from fields other than CorrectlySpelledTerms.
>
> In other words I don't think that anything is getting into TermsDictionary from CorrectlySpelledTerms...
>
> Be careful to remove the index between schema changes, just to be sure that you're not seeing old data.
>
> Best
> Erick
>
> On Mon, Aug 22, 2011 at 11:41 AM, Herman Kiefus <he...@angieslist.com> wrote:
>> That's what I thought, but my experiments show differently. In actuality:
>>
>> I have a number of fields that are of type "text" (the default as it is packaged).
>>
>> I have a type 'textCorrectlySpelled' that utilizes KeepWordFilterFactory in index-time analysis, using a file of terms which are known to be correctly spelled.
>>
>> I have a type 'textDictionary' that has no index-time analysis.
>>
>> I have the fields:
>> <field name="CorrectlySpelledTerms" type="textCorrectlySpelled"
>> indexed="false" stored="false" multiValued="true"/> <field
>> name="TermsDictionary" type="textDictionary" indexed="true"
>> stored="false" multiValued="true"/>
>>
>> I want 'TermsDictionary' to contain only those terms from some fields that are correctly spelled plus those terms from a couple other fields (CompanyName and ContactName) as is. I use several copyField directives as follows:
>>
>> <copyField source="Field1" dest="CorrectlySpelledTerms"/> <copyField
>> source="Field2" dest="CorrectlySpelledTerms"/> <copyField
>> source="Field3" dest="CorrectlySpelledTerms"/>
>>
>> <copyField source="Name" dest="TermsDictionary"/> <copyField
>> source="Contact" dest="TermsDictionary"/> <copyField source
>> ="CorrectlySpelledTerms" dest="TermsDictionary"/>
>>
>> If I query 'Field1' for a term that I know is misspelled (electical) it yields results.
>> If I query 'TermsDictionary' for the same term it yields no results.
>>
>> It would seem by these results that 'TermsDictionary' only contains those terms with misspellings stripped as a results of the text analysis on the field 'CorrectlySpelledTerms'.
>>
>> Asked another way, I think you can see what I'm getting at: a source for the spellchecker that only contains correct spelled terms plus proper names; should I have gone about this in a different way?
>>
>> -----Original Message-----
>> From: Stephen Duncan Jr [mailto:stephen.duncan@gmail.com]
>> Sent: Monday, August 22, 2011 9:30 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Text Analysis and copyField
>>
>> On Mon, Aug 22, 2011 at 9:25 AM, Herman Kiefus <he...@angieslist.com> wrote:
>>> Is my thinking correct?
>>>
>>> I have a field 'F1' of type 'T1' whose index time analysis employs the StopFilterFactory.
>>>
>>> I also have a field 'F2' of type 'T2' whose index time analysis does NOT employ the StopFilterFactory.
>>>
>>> There is a copyField directive source="F1" dest="F2"
>>>
>>> F2 will not contain any stop words because they were filtered out as F1 was populated.
>>>
>>
>> No, F2 will contain stop words. Copy fields does not process input through a chain, it sends the original content to each field and therefore analysis is totally independent.
>>
>> --
>> Stephen Duncan Jr
>> www.stephenduncanjr.com
>>
>
Re: Text Analysis and copyField
Posted by Erick Erickson <er...@gmail.com>.
Have you considered having two dictionaries and using ajax to query
them both and
intermingling the results in your suggestions? It'd be some work, but
I think it might
accomplish what you want.
Best
Erick
On Tue, Aug 23, 2011 at 1:48 PM, Herman Kiefus <he...@angieslist.com> wrote:
> To close, I found this article from Hoss: http://lucene.472066.n3.nabble.com/CopyField-into-another-CopyField-td3122408.html
>
> Since I cannot use one copyField directive to copy from another copyField's dest[ination], I cannot achieve what I desire: some terms that are subject to KeepWordFilterFactory and some that are not.
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Monday, August 22, 2011 1:16 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Text Analysis and copyField
>
> I suspect that the things going into TermsDictionary are from fields other than CorrectlySpelledTerms.
>
> In other words I don't think that anything is getting into TermsDictionary from CorrectlySpelledTerms...
>
> Be careful to remove the index between schema changes, just to be sure that you're not seeing old data.
>
> Best
> Erick
>
> On Mon, Aug 22, 2011 at 11:41 AM, Herman Kiefus <he...@angieslist.com> wrote:
>> That's what I thought, but my experiments show differently. In actuality:
>>
>> I have a number of fields that are of type "text" (the default as it is packaged).
>>
>> I have a type 'textCorrectlySpelled' that utilizes KeepWordFilterFactory in index-time analysis, using a file of terms which are known to be correctly spelled.
>>
>> I have a type 'textDictionary' that has no index-time analysis.
>>
>> I have the fields:
>> <field name="CorrectlySpelledTerms" type="textCorrectlySpelled"
>> indexed="false" stored="false" multiValued="true"/> <field
>> name="TermsDictionary" type="textDictionary" indexed="true"
>> stored="false" multiValued="true"/>
>>
>> I want 'TermsDictionary' to contain only those terms from some fields that are correctly spelled plus those terms from a couple other fields (CompanyName and ContactName) as is. I use several copyField directives as follows:
>>
>> <copyField source="Field1" dest="CorrectlySpelledTerms"/> <copyField
>> source="Field2" dest="CorrectlySpelledTerms"/> <copyField
>> source="Field3" dest="CorrectlySpelledTerms"/>
>>
>> <copyField source="Name" dest="TermsDictionary"/> <copyField
>> source="Contact" dest="TermsDictionary"/> <copyField source
>> ="CorrectlySpelledTerms" dest="TermsDictionary"/>
>>
>> If I query 'Field1' for a term that I know is misspelled (electical) it yields results.
>> If I query 'TermsDictionary' for the same term it yields no results.
>>
>> It would seem by these results that 'TermsDictionary' only contains those terms with misspellings stripped as a results of the text analysis on the field 'CorrectlySpelledTerms'.
>>
>> Asked another way, I think you can see what I'm getting at: a source for the spellchecker that only contains correct spelled terms plus proper names; should I have gone about this in a different way?
>>
>> -----Original Message-----
>> From: Stephen Duncan Jr [mailto:stephen.duncan@gmail.com]
>> Sent: Monday, August 22, 2011 9:30 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Text Analysis and copyField
>>
>> On Mon, Aug 22, 2011 at 9:25 AM, Herman Kiefus <he...@angieslist.com> wrote:
>>> Is my thinking correct?
>>>
>>> I have a field 'F1' of type 'T1' whose index time analysis employs the StopFilterFactory.
>>>
>>> I also have a field 'F2' of type 'T2' whose index time analysis does NOT employ the StopFilterFactory.
>>>
>>> There is a copyField directive source="F1" dest="F2"
>>>
>>> F2 will not contain any stop words because they were filtered out as F1 was populated.
>>>
>>
>> No, F2 will contain stop words. Copy fields does not process input through a chain, it sends the original content to each field and therefore analysis is totally independent.
>>
>> --
>> Stephen Duncan Jr
>> www.stephenduncanjr.com
>>
>
RE: Text Analysis and copyField
Posted by Herman Kiefus <he...@angieslist.com>.
To close, I found this article from Hoss: http://lucene.472066.n3.nabble.com/CopyField-into-another-CopyField-td3122408.html
Since I cannot use one copyField directive to copy from another copyField's dest[ination], I cannot achieve what I desire: some terms that are subject to KeepWordFilterFactory and some that are not.
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Monday, August 22, 2011 1:16 PM
To: solr-user@lucene.apache.org
Subject: Re: Text Analysis and copyField
I suspect that the things going into TermsDictionary are from fields other than CorrectlySpelledTerms.
In other words I don't think that anything is getting into TermsDictionary from CorrectlySpelledTerms...
Be careful to remove the index between schema changes, just to be sure that you're not seeing old data.
Best
Erick
On Mon, Aug 22, 2011 at 11:41 AM, Herman Kiefus <he...@angieslist.com> wrote:
> That's what I thought, but my experiments show differently. In actuality:
>
> I have a number of fields that are of type "text" (the default as it is packaged).
>
> I have a type 'textCorrectlySpelled' that utilizes KeepWordFilterFactory in index-time analysis, using a file of terms which are known to be correctly spelled.
>
> I have a type 'textDictionary' that has no index-time analysis.
>
> I have the fields:
> <field name="CorrectlySpelledTerms" type="textCorrectlySpelled"
> indexed="false" stored="false" multiValued="true"/> <field
> name="TermsDictionary" type="textDictionary" indexed="true"
> stored="false" multiValued="true"/>
>
> I want 'TermsDictionary' to contain only those terms from some fields that are correctly spelled plus those terms from a couple other fields (CompanyName and ContactName) as is. I use several copyField directives as follows:
>
> <copyField source="Field1" dest="CorrectlySpelledTerms"/> <copyField
> source="Field2" dest="CorrectlySpelledTerms"/> <copyField
> source="Field3" dest="CorrectlySpelledTerms"/>
>
> <copyField source="Name" dest="TermsDictionary"/> <copyField
> source="Contact" dest="TermsDictionary"/> <copyField source
> ="CorrectlySpelledTerms" dest="TermsDictionary"/>
>
> If I query 'Field1' for a term that I know is misspelled (electical) it yields results.
> If I query 'TermsDictionary' for the same term it yields no results.
>
> It would seem by these results that 'TermsDictionary' only contains those terms with misspellings stripped as a results of the text analysis on the field 'CorrectlySpelledTerms'.
>
> Asked another way, I think you can see what I'm getting at: a source for the spellchecker that only contains correct spelled terms plus proper names; should I have gone about this in a different way?
>
> -----Original Message-----
> From: Stephen Duncan Jr [mailto:stephen.duncan@gmail.com]
> Sent: Monday, August 22, 2011 9:30 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Text Analysis and copyField
>
> On Mon, Aug 22, 2011 at 9:25 AM, Herman Kiefus <he...@angieslist.com> wrote:
>> Is my thinking correct?
>>
>> I have a field 'F1' of type 'T1' whose index time analysis employs the StopFilterFactory.
>>
>> I also have a field 'F2' of type 'T2' whose index time analysis does NOT employ the StopFilterFactory.
>>
>> There is a copyField directive source="F1" dest="F2"
>>
>> F2 will not contain any stop words because they were filtered out as F1 was populated.
>>
>
> No, F2 will contain stop words. Copy fields does not process input through a chain, it sends the original content to each field and therefore analysis is totally independent.
>
> --
> Stephen Duncan Jr
> www.stephenduncanjr.com
>
Re: Text Analysis and copyField
Posted by Erick Erickson <er...@gmail.com>.
I suspect that the things going into TermsDictionary are from fields other than
CorrectlySpelledTerms.
In other words I don't think that anything is getting into TermsDictionary from
CorrectlySpelledTerms...
Be careful to remove the index between schema changes, just to be sure that
you're not seeing old data.
Best
Erick
On Mon, Aug 22, 2011 at 11:41 AM, Herman Kiefus <he...@angieslist.com> wrote:
> That's what I thought, but my experiments show differently. In actuality:
>
> I have a number of fields that are of type "text" (the default as it is packaged).
>
> I have a type 'textCorrectlySpelled' that utilizes KeepWordFilterFactory in index-time analysis, using a file of terms which are known to be correctly spelled.
>
> I have a type 'textDictionary' that has no index-time analysis.
>
> I have the fields:
> <field name="CorrectlySpelledTerms" type="textCorrectlySpelled" indexed="false" stored="false" multiValued="true"/>
> <field name="TermsDictionary" type="textDictionary" indexed="true" stored="false" multiValued="true"/>
>
> I want 'TermsDictionary' to contain only those terms from some fields that are correctly spelled plus those terms from a couple other fields (CompanyName and ContactName) as is. I use several copyField directives as follows:
>
> <copyField source="Field1" dest="CorrectlySpelledTerms"/>
> <copyField source="Field2" dest="CorrectlySpelledTerms"/>
> <copyField source="Field3" dest="CorrectlySpelledTerms"/>
>
> <copyField source="Name" dest="TermsDictionary"/>
> <copyField source="Contact" dest="TermsDictionary"/>
> <copyField source ="CorrectlySpelledTerms" dest="TermsDictionary"/>
>
> If I query 'Field1' for a term that I know is misspelled (electical) it yields results.
> If I query 'TermsDictionary' for the same term it yields no results.
>
> It would seem by these results that 'TermsDictionary' only contains those terms with misspellings stripped as a results of the text analysis on the field 'CorrectlySpelledTerms'.
>
> Asked another way, I think you can see what I'm getting at: a source for the spellchecker that only contains correct spelled terms plus proper names; should I have gone about this in a different way?
>
> -----Original Message-----
> From: Stephen Duncan Jr [mailto:stephen.duncan@gmail.com]
> Sent: Monday, August 22, 2011 9:30 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Text Analysis and copyField
>
> On Mon, Aug 22, 2011 at 9:25 AM, Herman Kiefus <he...@angieslist.com> wrote:
>> Is my thinking correct?
>>
>> I have a field 'F1' of type 'T1' whose index time analysis employs the StopFilterFactory.
>>
>> I also have a field 'F2' of type 'T2' whose index time analysis does NOT employ the StopFilterFactory.
>>
>> There is a copyField directive source="F1" dest="F2"
>>
>> F2 will not contain any stop words because they were filtered out as F1 was populated.
>>
>
> No, F2 will contain stop words. Copy fields does not process input through a chain, it sends the original content to each field and therefore analysis is totally independent.
>
> --
> Stephen Duncan Jr
> www.stephenduncanjr.com
>
RE: Text Analysis and copyField
Posted by Herman Kiefus <he...@angieslist.com>.
That's what I thought, but my experiments show differently. In actuality:
I have a number of fields that are of type "text" (the default as it is packaged).
I have a type 'textCorrectlySpelled' that utilizes KeepWordFilterFactory in index-time analysis, using a file of terms which are known to be correctly spelled.
I have a type 'textDictionary' that has no index-time analysis.
I have the fields:
<field name="CorrectlySpelledTerms" type="textCorrectlySpelled" indexed="false" stored="false" multiValued="true"/>
<field name="TermsDictionary" type="textDictionary" indexed="true" stored="false" multiValued="true"/>
I want 'TermsDictionary' to contain only those terms from some fields that are correctly spelled plus those terms from a couple other fields (CompanyName and ContactName) as is. I use several copyField directives as follows:
<copyField source="Field1" dest="CorrectlySpelledTerms"/>
<copyField source="Field2" dest="CorrectlySpelledTerms"/>
<copyField source="Field3" dest="CorrectlySpelledTerms"/>
<copyField source="Name" dest="TermsDictionary"/>
<copyField source="Contact" dest="TermsDictionary"/>
<copyField source ="CorrectlySpelledTerms" dest="TermsDictionary"/>
If I query 'Field1' for a term that I know is misspelled (electical) it yields results.
If I query 'TermsDictionary' for the same term it yields no results.
It would seem by these results that 'TermsDictionary' only contains those terms with misspellings stripped as a results of the text analysis on the field 'CorrectlySpelledTerms'.
Asked another way, I think you can see what I'm getting at: a source for the spellchecker that only contains correct spelled terms plus proper names; should I have gone about this in a different way?
-----Original Message-----
From: Stephen Duncan Jr [mailto:stephen.duncan@gmail.com]
Sent: Monday, August 22, 2011 9:30 AM
To: solr-user@lucene.apache.org
Subject: Re: Text Analysis and copyField
On Mon, Aug 22, 2011 at 9:25 AM, Herman Kiefus <he...@angieslist.com> wrote:
> Is my thinking correct?
>
> I have a field 'F1' of type 'T1' whose index time analysis employs the StopFilterFactory.
>
> I also have a field 'F2' of type 'T2' whose index time analysis does NOT employ the StopFilterFactory.
>
> There is a copyField directive source="F1" dest="F2"
>
> F2 will not contain any stop words because they were filtered out as F1 was populated.
>
No, F2 will contain stop words. Copy fields does not process input through a chain, it sends the original content to each field and therefore analysis is totally independent.
--
Stephen Duncan Jr
www.stephenduncanjr.com
Re: Text Analysis and copyField
Posted by Stephen Duncan Jr <st...@gmail.com>.
On Mon, Aug 22, 2011 at 9:25 AM, Herman Kiefus <he...@angieslist.com> wrote:
> Is my thinking correct?
>
> I have a field 'F1' of type 'T1' whose index time analysis employs the StopFilterFactory.
>
> I also have a field 'F2' of type 'T2' whose index time analysis does NOT employ the StopFilterFactory.
>
> There is a copyField directive source="F1" dest="F2"
>
> F2 will not contain any stop words because they were filtered out as F1 was populated.
>
No, F2 will contain stop words. Copy fields does not process input
through a chain, it sends the original content to each field and
therefore analysis is totally independent.
--
Stephen Duncan Jr
www.stephenduncanjr.com