You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by "Flowerday, Matthew J" <ma...@gb.unisys.com> on 2021/07/07 19:17:24 UTC
Solr Phonetic Search funny
Hi There
I have just noticed a funny with DoubleMetaphone phonetic searching
involving records with certain values in date fields causing a phonetic
match to be returned for that record.
I was searching for the word 'main' and I found results being returned with
nothing being highlighted via the unified highlighter. I tracked the issue
down to these date fields holding this values
"statementDate_dtr":"2019-10-28T00:00:00Z",
"statementDate_dt":"28/10/2019",
Now the word 'main' has a phonetic value of 'MN' (according to the Solr
Admin Tool analysis feature). Searching for either 'main' or 'mn' in the
Admin tool returns a match on the record with these date values.
The fields are configured as
<dynamicField name="*_dt" type="text_en_uleaf" indexed="true"
stored="true"/>
<dynamicField name="*_dtr" type="pdate" indexed="true" stored="true"/>
I suspect the issue is down to the _dtr field as the _dt field is basically
a string field.
If I place the text 2019-10-28T00:00:00Z in a standard string field on
another record the phonetic search does not match for 'main' on that record
which does seem to point to the issue being related to a date field.
If I then update the record and change the date to be say
"takenDate_dtr":"2021-07-07T00:00:00Z",
"takenDate_dt":"07/07/2021",
Then a search for 'main' does not find a phonetic match.
I am using Solr 8.8.1 and Solr 8.9.0 and the issue appears in both versions.
I was wondering if anyone has seen this before?
Many Thanks
Matthew
Matthew Flowerday | Consultant | ULEAF
Unisys | 01908 774830| <ma...@unisys.com>
matthew.flowerday@unisys.com
Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17
8LX
<http://www.unisys.com/>
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is for use only by the intended recipient. If you received this
in error, please contact the sender and delete the e-mail and its
attachments from all devices.
<http://www.linkedin.com/company/unisys> <http://twitter.com/unisyscorp>
<http://www.youtube.com/theunisyschannel>
<http://www.facebook.com/unisyscorp> <https://vimeo.com/unisys>
<http://blogs.unisys.com/>
RE: Solr Phonetic Search funny
Posted by "Flowerday, Matthew J" <ma...@gb.unisys.com>.
Hi Alex
That sorted it! Thank you so much.
On Solr 8.9.0 I had to set update.autoCreateFields to be false in the add-unknown-fields-to-the-schema below:
<updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:false}"
processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date">
<processor class="solr.LogUpdateProcessorFactory"/>
<processor class="solr.DistributedUpdateProcessorFactory"/>
<processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>
And that sorted it.
Many thanks again for your help!
Matthew
Matthew Flowerday | Consultant | ULEAF
Unisys | 01908 774830| matthew.flowerday@unisys.com <ma...@unisys.com>
Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX
<http://www.unisys.com/>
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.
<http://www.linkedin.com/company/unisys> <http://twitter.com/unisyscorp> <http://www.youtube.com/theunisyschannel> <http://www.facebook.com/unisyscorp> <https://vimeo.com/unisys> <http://blogs.unisys.com/>
From: Alexandre Rafalovitch <ar...@gmail.com>
Sent: 08 July 2021 00:13
To: users@solr.apache.org
Subject: Re: Solr Phonetic Search funny
EXTERNAL EMAIL - Be cautious of all links and attachments.
Check if you have "schemaless" processing enabled.
If yes, then it is most likely because schemaless pipeline converted your date to real date during preprocessong, then when the schema said "it is a text/string", the actual Date got serialized back into a string, using a local formatting (probably a "date long").
If you check indexed tokens in Admin UI for that field, it may give you a confirmation.
See https://www.outerthoughts.com/2016/04/solr-5-puzzle-magic-date-answer/
Regards,
Alex
On Wed., Jul. 7, 2021, 3:17 p.m. Flowerday, Matthew J, <matthew.flowerday@gb.unisys.com <ma...@gb.unisys.com> > wrote:
Hi There
I have just noticed a funny with DoubleMetaphone phonetic searching involving records with certain values in date fields causing a phonetic match to be returned for that record.
I was searching for the word ‘main’ and I found results being returned with nothing being highlighted via the unified highlighter. I tracked the issue down to these date fields holding this values
"statementDate_dtr":"2019-10-28T00:00:00Z",
"statementDate_dt":"28/10/2019",
Now the word ‘main’ has a phonetic value of ‘MN’ (according to the Solr Admin Tool analysis feature). Searching for either ‘main’ or ‘mn’ in the Admin tool returns a match on the record with these date values.
The fields are configured as
<dynamicField name="*_dt" type="text_en_uleaf" indexed="true" stored="true"/>
<dynamicField name="*_dtr" type="pdate" indexed="true" stored="true"/>
I suspect the issue is down to the _dtr field as the _dt field is basically a string field.
If I place the text 2019-10-28T00:00:00Z in a standard string field on another record the phonetic search does not match for ‘main’ on that record which does seem to point to the issue being related to a date field.
If I then update the record and change the date to be say
"takenDate_dtr":"2021-07-07T00:00:00Z",
"takenDate_dt":"07/07/2021",
Then a search for ‘main’ does not find a phonetic match.
I am using Solr 8.8.1 and Solr 8.9.0 and the issue appears in both versions.
I was wondering if anyone has seen this before?
Many Thanks
Matthew
Matthew Flowerday | Consultant | ULEAF
Unisys | 01908 774830| matthew.flowerday@unisys.com <ma...@unisys.com>
Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX
<http://www.unisys.com/>
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.
<http://www.linkedin.com/company/unisys> <http://twitter.com/unisyscorp> <http://www.youtube.com/theunisyschannel> <http://www.facebook.com/unisyscorp> <https://vimeo.com/unisys> <http://blogs.unisys.com/>
Re: Solr Phonetic Search funny
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Check if you have "schemaless" processing enabled.
If yes, then it is most likely because schemaless pipeline converted your
date to real date during preprocessong, then when the schema said "it is a
text/string", the actual Date got serialized back into a string, using a
local formatting (probably a "date long").
If you check indexed tokens in Admin UI for that field, it may give you a
confirmation.
See https://www.outerthoughts.com/2016/04/solr-5-puzzle-magic-date-answer/
Regards,
Alex
On Wed., Jul. 7, 2021, 3:17 p.m. Flowerday, Matthew J, <
matthew.flowerday@gb.unisys.com> wrote:
> Hi There
>
>
>
> I have just noticed a funny with DoubleMetaphone phonetic searching
> involving records with certain values in date fields causing a phonetic
> match to be returned for that record.
>
>
>
> I was searching for the word ‘main’ and I found results being returned
> with nothing being highlighted via the unified highlighter. I tracked the
> issue down to these date fields holding this values
>
>
>
> "*statementDate_dtr*":"2019-10-28T00:00:00Z",
>
> "*statementDate_dt*":"28/10/2019",
>
>
>
> Now the word ‘main’ has a phonetic value of ‘MN’ (according to the Solr
> Admin Tool analysis feature). Searching for either ‘main’ or ‘mn’ in the
> Admin tool returns a match on the record with these date values.
>
>
>
> The fields are configured as
>
>
>
> <dynamicField name="*_dt" type="text_en_uleaf" indexed="true"
> stored="true"/>
>
> <dynamicField name="*_dtr" type="pdate" indexed="true" stored="true"/>
>
>
>
> I suspect the issue is down to the _dtr field as the _dt field is
> basically a string field.
>
>
>
> If I place the text 2019-10-28T00:00:00Z in a standard string field on
> another record the phonetic search does not match for ‘main’ on that record
> which does seem to point to the issue being related to a date field.
>
>
>
> If I then update the record and change the date to be say
>
>
>
> "*takenDate_dtr*":"2021-07-07T00:00:00Z",
>
> "*takenDate_dt*":"07/07/2021",
>
>
>
> Then a search for ‘main’ does not find a phonetic match.
>
>
>
> I am using Solr 8.8.1 and Solr 8.9.0 and the issue appears in both
> versions.
>
>
>
> I was wondering if anyone has seen this before?
>
>
>
> Many Thanks
>
>
>
> Matthew
>
>
>
> *Matthew Flowerday* | Consultant | ULEAF
>
> Unisys | 01908 774830| matthew.flowerday@unisys.com
>
> Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17
> 8LX
>
>
>
> [image: unisys_logo] <http://www.unisys.com/>
>
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all devices.
>
> [image: Grey_LI] <http://www.linkedin.com/company/unisys> [image:
> Grey_TW] <http://twitter.com/unisyscorp> [image: Grey_YT]
> <http://www.youtube.com/theunisyschannel>[image: Grey_FB]
> <http://www.facebook.com/unisyscorp>[image: Grey_Vimeo]
> <https://vimeo.com/unisys>[image: Grey_UB] <http://blogs.unisys.com/>
>
>
>