You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Shamik Bandopadhyay <sh...@gmail.com> on 2015/05/18 23:41:15 UTC
Issue with German search
Hi,
I'm having an issue with searching a term in german. Here's the keyword(s)
I'm trying to search --> Software und Downloads
I've a document indexed in German with the same title --> Software und
Downloads
I'm expecting that the search on "Software und Downloads" will return this
document, unfortunately it's not happening.
Here's my sample test scenario from my local machine.
In schema, I've defined these three fields.
<field name="title_deu" type="adsktext_deu" indexed="true" stored="true"
multiValued="true" />
<field name="name_deu" type="adsktext_deu" indexed="true" stored="true"
termVectors="true" termPositions="true" termOffsets="true"/>
<field name="description_deu" type="adsktext_deu" indexed="true"
stored="true" termVectors="true" termPositions="true" termOffsets="true"/>
Field Type definition :
<!-- German language specific definitions -->
<fieldType name="adsktext_deu" class="solr.TextField"
positionIncrementGap="100" autoGeneratePhraseQueries="true">
<analyzer type="index">
<charFilter class="solr.MappingCharFilterFactory"
mapping="mapping-ISOLatin1Accent.txt" />
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_de.txt" format="snowball" />
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
generateNumberParts="1" catenateWords="1" catenateNumbers="1"
catenateAll="0" splitOnCaseChange="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.DictionaryCompoundWordTokenFilterFactory"
dictionary="lang/dictionary_de.txt" />
<filter class="solr.GermanNormalizationFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="German2"/>
</analyzer>
<analyzer type="query">
<charFilter class="solr.MappingCharFilterFactory"
mapping="mapping-ISOLatin1Accent.txt" />
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="lang/stopwords_de.txt" format="snowball" />
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
generateNumberParts="1" catenateWords="1" catenateNumbers="1"
catenateAll="0" splitOnCaseChange="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.DictionaryCompoundWordTokenFilterFactory"
dictionary="lang/dictionary_de.txt" />
<filter class="solr.GermanNormalizationFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="German2"/>
</analyzer>
</fieldType>
When I ran a sample analysis of "Software und Downloads", the term is
indexed as
softwar soft oft download ad
During query, it's getting searched as
softwar download
Not sure, why it's not returning the document.
Here's the sample data indexed through solr.xml under example docs.
<doc>
<field name="id">999912234!SOLR11092212</field>
<field name="name_deu">Test Name</field>
<field name="title_deu">Software und Downloads</field>
<field name="description_deu">div#actcontain { width: 100%; min-width:
220px; display: block; float: left; padding: 0 8px 0 0; } div#actcopy {
width: 48%; min-width: 230px; min-height: 120px; float: left; display:
inline-block; padding: 0 28px 0 0; margin: 10px 0 0 0; overflow: hidden; }
Häufige ThemenDownload-Verfahren im Autodesk-KontoDownload-Verfahren für
Education Community (Schüler, Studenten und Lehrkräfte)Suchen von Service
Packs, Hotfixes und SprachpaketenSoftware-Lizenzen im Autodesk-Store
kaufenSuchen kostenloser Testversion-Downloads
Download-VerfahrenHerunterladen von Software aus verschiedenen
Speicherorten, abhängig von Ihrem Konto oder dem Subscription-TypenNutzung
am Heimarbeitsplatz für AbonnentenDesktop Subscription können lizenzierte
Software zur Verwendung auf ihrem Computer zu Hause
erhaltenProdukterweiterungen für AbonnentenExklusiver Zugriff auf die
neueste Software für einige Autodesk-ProdukteBestellen einer Software-DVDSo
bestellen Sie eine DVD oder einen USB-Stick für Ihre SoftwareAktuelle
Versionen für AbonnentenSubscription-Kunden haben Zugriff auf
Produkt-Updates, die während der Vertragslaufzeit verfügbar
sind.VorgängerversionenErfahren Sie, wie Sie eine Vorgängerversion Ihrer
Autodesk-Software erhaltenSprachoptionenHerunterladen der lizenzierten
Software in einer anderen Sprache oder Erhalten eines Sprachpakets.
</field>
<field name="author">Bob</field>
</doc>
Any pointers will be appreciated.
-Thanks,
Shamik
Re: Issue with German search
Posted by Doug Turnbull <dt...@opensourceconnections.com>.
No prob, easy mistake to make :)
On Tue, May 19, 2015 at 2:14 PM, shamik <sh...@gmail.com> wrote:
> Thanks a ton Doug, I should have figured this out, pretty stupid of me.
>
> Appreciate your help.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Issue-with-German-search-tp4206104p4206357.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
*Doug Turnbull **| *Search Relevance Consultant | OpenSource Connections,
LLC | 240.476.9983 | http://www.opensourceconnections.com
Author: Relevant Search <http://manning.com/turnbull> from Manning
Publications
This e-mail and all contents, including attachments, is considered to be
Company Confidential unless explicitly stated otherwise, regardless
of whether attachments are marked as such.
Re: Issue with German search
Posted by shamik <sh...@gmail.com>.
Thanks a ton Doug, I should have figured this out, pretty stupid of me.
Appreciate your help.
--
View this message in context: http://lucene.472066.n3.nabble.com/Issue-with-German-search-tp4206104p4206357.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Issue with German search
Posted by Doug Turnbull <dt...@opensourceconnections.com>.
I see the problem, instead of using debugQuery directly you might want to
check out splainer (http://splainer.io) it can help detect these sorts of
relevance problems. For example, in your case you just need to wrap the
search in parenthesis.
There's a difference between text:foo bar and text:(foo bar). Without the
parethesis the "bar" goes to the default search field. Instead of
q=title_deu:Software%20und%20Downloads try q=title_deu:(Software%
20und%20Downloads)
This example shows your problem using a different Solr instance, you can
see "hunting" matches are happening in text, not the catch_line field
http://splainer.io/#?solr=http:%2F%2Fsolr.quepid.com%2Fsolr%2Fstatedecoded%2Fselect%3Fq%3Dcatch_line:deer%20hunting
wrap in parens and viola, both matches go to the catch_line field:
http://splainer.io/#?solr=http:%2F%2Fsolr.quepid.com%2Fsolr%2Fstatedecoded%2Fselect%3Fq%3Dcatch_line:(deer%20hunting)
Hope that helps
-Doug
On Tue, May 19, 2015 at 1:45 PM, shamik <sh...@gmail.com> wrote:
> Thanks Doug. I'm using eDismax
>
> Here's my Solr query :
>
>
> http://localhost:8983/solr/testhandlerdeu?debugQuery=true&q=title_deu:Software%20und%20Downloads
>
> Here's my request handler.
>
> <requestHandler name="/testhandlerdeu" class="solr.SearchHandler">
> <lst name="defaults">
> <str name="echoParams">explicit</str>
> <float name="tie">0.01</float>
> <str name="wt">velocity</str>
> <str name="v.template">browse</str>
> <str
> name="v.contentType">text/html;charset=UTF-8</str>
> <str name="v.layout">layout</str>
> <str name="v.channel">testhandler</str>
> <str name="title">Test Request Handler German</str>
>
> <str name="defType">edismax</str>
> <str name="q.op">AND</str>
>
> <str name="q.alt">*:*</str>
> <str name="rows">15</str>
> <str name="fl">*,score</str>
> <str name="qf">name_deu^1.2 title_deu^10.0
> description_deu^5.0 </str>
> <str name="df">text_deu</str>
>
>
> <str name="facet">on</str>
> <str name="facet.mincount">1</str>
> <str name="facet.limit">-1</str>
> <str name="facet.sort">index</str>
> <str name="facet.method">enum</str>
> <str name="facet.field">cat</str>
> <str name="facet.field">manu_exact</str>
> <str name="facet.field">content_type</str>
> <str name="facet.field">author</str>
>
>
> <str name="hl">true</str>
> <str name="hl.tag.pre"></str>
> <str name="hl.tag.post"></str>
> <str name="hl.fl">name subject description_deu
> name_deu title_deu</str>
> <str name="hl.encoder">html</str>
> <str name="f.subject.hl.fragsize">20</str>
> <str name="f.description_fra.hl.fragsize">20</str>
> <str name="f.name_fra.hl.fragsize">20</str>
> <str name="hl.usePhraseHighlighter">false</str>
> <str name="hl.useFastVectorHighlighter">true</str>
> <str name="hl.boundaryScanner">breakIterator</str>
> <str name="hl.bs.type">SENTENCE</str>
>
>
> <str name="spellcheck">true</str>
> <str name="spellcheck.dictionary">default</str>
> <str name="spellcheck.collate">true</str>
> <str name="spellcheck.onlyMorePopular">false</str>
> <str name="spellcheck.extendedResults">false</str>
> <str name="spellcheck.count">1</str>
>
> </lst>
> <arr name="last-components">
> <str>spellcheck</str>
> </arr>
> </requestHandler>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Issue-with-German-search-tp4206104p4206341.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
*Doug Turnbull **| *Search Relevance Consultant | OpenSource Connections,
LLC | 240.476.9983 | http://www.opensourceconnections.com
Author: Relevant Search <http://manning.com/turnbull> from Manning
Publications
This e-mail and all contents, including attachments, is considered to be
Company Confidential unless explicitly stated otherwise, regardless
of whether attachments are marked as such.
Re: Issue with German search
Posted by shamik <sh...@gmail.com>.
Thanks Doug. I'm using eDismax
Here's my Solr query :
http://localhost:8983/solr/testhandlerdeu?debugQuery=true&q=title_deu:Software%20und%20Downloads
Here's my request handler.
<requestHandler name="/testhandlerdeu" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<float name="tie">0.01</float>
<str name="wt">velocity</str>
<str name="v.template">browse</str>
<str name="v.contentType">text/html;charset=UTF-8</str>
<str name="v.layout">layout</str>
<str name="v.channel">testhandler</str>
<str name="title">Test Request Handler German</str>
<str name="defType">edismax</str>
<str name="q.op">AND</str>
<str name="q.alt">*:*</str>
<str name="rows">15</str>
<str name="fl">*,score</str>
<str name="qf">name_deu^1.2 title_deu^10.0 description_deu^5.0 </str>
<str name="df">text_deu</str>
<str name="facet">on</str>
<str name="facet.mincount">1</str>
<str name="facet.limit">-1</str>
<str name="facet.sort">index</str>
<str name="facet.method">enum</str>
<str name="facet.field">cat</str>
<str name="facet.field">manu_exact</str>
<str name="facet.field">content_type</str>
<str name="facet.field">author</str>
<str name="hl">true</str>
<str name="hl.tag.pre"></str>
<str name="hl.tag.post"></str>
<str name="hl.fl">name subject description_deu name_deu title_deu</str>
<str name="hl.encoder">html</str>
<str name="f.subject.hl.fragsize">20</str>
<str name="f.description_fra.hl.fragsize">20</str>
<str name="f.name_fra.hl.fragsize">20</str>
<str name="hl.usePhraseHighlighter">false</str>
<str name="hl.useFastVectorHighlighter">true</str>
<str name="hl.boundaryScanner">breakIterator</str>
<str name="hl.bs.type">SENTENCE</str>
<str name="spellcheck">true</str>
<str name="spellcheck.dictionary">default</str>
<str name="spellcheck.collate">true</str>
<str name="spellcheck.onlyMorePopular">false</str>
<str name="spellcheck.extendedResults">false</str>
<str name="spellcheck.count">1</str>
</lst>
<arr name="last-components">
<str>spellcheck</str>
</arr>
</requestHandler>
--
View this message in context: http://lucene.472066.n3.nabble.com/Issue-with-German-search-tp4206104p4206341.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Issue with German search
Posted by Doug Turnbull <dt...@opensourceconnections.com>.
How are you searching shamik? What query parser are you using? Perhaps you
could share a sample Solr URL?
Cheers,
-Doug
On Tue, May 19, 2015 at 11:11 AM, shamik <sh...@gmail.com> wrote:
> Anyone ?
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Issue-with-German-search-tp4206104p4206306.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
*Doug Turnbull **| *Search Relevance Consultant | OpenSource Connections,
LLC | 240.476.9983 | http://www.opensourceconnections.com
Author: Relevant Search <http://manning.com/turnbull> from Manning
Publications
This e-mail and all contents, including attachments, is considered to be
Company Confidential unless explicitly stated otherwise, regardless
of whether attachments are marked as such.
Re: Issue with German search
Posted by shamik <sh...@gmail.com>.
Anyone ?
--
View this message in context: http://lucene.472066.n3.nabble.com/Issue-with-German-search-tp4206104p4206306.html
Sent from the Solr - User mailing list archive at Nabble.com.