You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Alejandro Garza Gonzalez <al...@itesm.mx> on 2008/08/12 01:22:53 UTC
Best way to index without diacritics
I have utf-8 content that I wat to index, however I want searches
without diacritics to return results.
For example, a document with the words "nino en mexico" should return
results like a document with the phrase "Niño en México".
Ideally, exact diacritic matches should score higher (searching for
"niño" exactly should make a document with "niño" score higher than a
document with "nino")
Any pointers on how to do this? I found about the
/solr/.ISOLatin1AccentFilterFactory but it seems to only strip
diacritics from iso-latin characters. How about UTF diacritics?
--
_________________ ___ _ _ _ _ _ _ _
*Ing. Alejandro Garza González*
Director, Tecnología e Innovación, Biblioteca
Tecnológico de Monterrey, Campus Monterrey
Tel.: 52(81) 8358-1400 ext. 4037 Fax: 52(81) 8328-4067
Enlace Intercampus: 80 689 4037
http://biblioteca.mty.itesm.mx
El contenido de este mensaje de datos no se considera oferta, propuesta
o acuerdo, sino hasta que sea confirmado en documento por escrito que
contenga la firma autógrafa del apoderado legal del ITESM. El contenido
de este mensaje de datos es confidencial y se entiende dirigido y para
uso exclusivo del destinatario, por lo que no podrá distribuirse y/o
difundirse por ningún medio sin la previa autorización del emisor
original. Si usted no es el destinatario, se le prohíbe su utilización
total o parcial para cualquier fin.
The content of this data transmission must not be considered an offer,
proposal, understanding or agreement unless it is confirmed in a
document signed by a legal representative of ITESM. The content of this
data transmission is confidential and is intended to be delivered only
to the addressees. Therefore, it shall not be distributed and/or
disclosed through any means without the authorization of the original
sender. If you are not the addressee, you are forbidden from using it,
either totally or partially, for any purpose.
Re: Best way to index without diacritics
Posted by lupiss <lu...@hotmail.com>.
hola de nuevo!
gracias por la yuda, ya pude solucionar lo de los acentos :D
por si a alguien le sirve el tip, todo está en poner en el schema.xml la
clase isolatin mas o menos así:
hello again!
thanks for the help, I could solve it for the accents: D
if someone helps the tip, everything is put in the schema.xml ISOLATINO
class more or less like this:
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<!-- in this example, we will only use synonyms at query time
<filter class="solr.SynonymFilterFactory"
synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
-->
<filter class="solr.ISOLatin1AccentFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPorterFilterFactory"
protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.ISOLatin1AccentFilterFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="0"
catenateNumbers="0" catenateAll="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPorterFilterFactory"
protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>
además los archivos xml que se van a indexar deberán estar codificados en
utf-8, en mi caso uso el solr integrado a una aplicación por lo cual ahora
tengo que ver como enviar la cadena a indexar con codificación utf-8 ,
alguien sabe cómo hacerlo?
xml files also to be indexed must be encoded in utf-8 in my case I use the
integrated application SOLR so now I see how to send the string with index
utf-8 encoding, someone knows how to do ?
--
View this message in context: http://www.nabble.com/Best-way-to-index-without-diacritics-tp18935599p23189640.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Best way to index without diacritics
Posted by wi...@gmail.com.
Amigo!
Viva Solr :)
Sent from my BlackBerry device on the Rogers Wireless Network
-----Original Message-----
From: lupiss <lu...@hotmail.com>
Date: Tue, 21 Apr 2009 16:17:04
To: <so...@lucene.apache.org>
Subject: Re: Best way to index without diacritics
hola, gracias por contestar. sí, yo creo que esa es la clase que me servirá,
pero no sé cómo implementarla, podrías decirme si tu ya la haz usado, y si
es así, decirme qué líneas incluíste en el schema.xml, en el config.xml, qué
.jar adjuntaste, etc, todos los detalles, o incluso si tienes un ejemplo que
puedas escribir en el foro por favor, se me hace difícil porque además de
que soy nueva en solr, toda la información está en inglés y pués yo no lo
entiendo muy bien que digamos :(
gracias nuevamente
hi, thanks for reply. yes, I believe that is the class that will serve me,
but I do not know how to implement it, and you could tell me if the beam
used, and if so, tell me what lines included in the schema.xml in
config.xml, which . jar attached, etc, all the details, or even if you have
an example that you can write in the forum please, I also made it difficult
because I am new to SOLR, all information is in English and then I do not
understand very well to say: (
thanks again
--
View this message in context: http://www.nabble.com/Best-way-to-index-without-diacritics-tp18935599p23166430.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Best way to index without diacritics
Posted by lupiss <lu...@hotmail.com>.
hola, gracias por contestar. sí, yo creo que esa es la clase que me servirá,
pero no sé cómo implementarla, podrías decirme si tu ya la haz usado, y si
es así, decirme qué líneas incluíste en el schema.xml, en el config.xml, qué
.jar adjuntaste, etc, todos los detalles, o incluso si tienes un ejemplo que
puedas escribir en el foro por favor, se me hace difícil porque además de
que soy nueva en solr, toda la información está en inglés y pués yo no lo
entiendo muy bien que digamos :(
gracias nuevamente
hi, thanks for reply. yes, I believe that is the class that will serve me,
but I do not know how to implement it, and you could tell me if the beam
used, and if so, tell me what lines included in the schema.xml in
config.xml, which . jar attached, etc, all the details, or even if you have
an example that you can write in the forum please, I also made it difficult
because I am new to SOLR, all information is in English and then I do not
understand very well to say: (
thanks again
--
View this message in context: http://www.nabble.com/Best-way-to-index-without-diacritics-tp18935599p23166430.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Best way to index without diacritics
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Does this help: http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters?highlight=(isolatin)#head-4ebf7aea23b3d6d34a1f8314f9de17334a3e2fac
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: lupiss <lu...@hotmail.com>
> To: solr-user@lucene.apache.org
> Sent: Tuesday, April 21, 2009 12:22:08 PM
> Subject: Re: Best way to index without diacritics
>
>
> Hola!
>
> Yo también tengo el mismo problema, ya tengo mis índices de mis documentos
> cambie el charset a iso-8859-1 y ya pude ver las ñ y acentos, ahora ligué el
> buscador a mi aplicación y desde páginas jsp se hacen búsquedas, el problema
> es que cuando el usuario escribe en el text que pide el parámetro de la
> consulta debe escribir los acentos, y eso es justamente lo que no quiero,
> leí un poco y creo que ocupo incluir la clase isolatin o algo así, pero no
> sé bien como, primero se que la debo incluir en el config.xml o en el
> schema.xml pero también debo bajar un jar o algo así de la clase isolatin? o
> que debo hacer no sé... por favor alguna ayuda!!!? me estoy volviendo loca!
> :(
>
>
> Hello!
>
> I too have the same problem as I have my documents indexes change charset to
> iso-8859-1, and I could see ñ and accents, now linking the browser to my
> application from jsp pages are searched, the problem is that when the user
> types in the text that requests the parameter of the query must write
> accents, and this is what we do not want to, I read a bit and I think the
> deal include ISOLATINO class or something, but I do not know as well, first
> is that I include in the config.xml or the schema.xml but I download a jar
> or something of the kind ISOLATINO? or that I do not know ... please some
> help !!!?. I'm going crazy! : (
> --
> View this message in context:
> http://www.nabble.com/Best-way-to-index-without-diacritics-tp18935599p23159812.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Best way to index without diacritics
Posted by lupiss <lu...@hotmail.com>.
Hola!
Yo también tengo el mismo problema, ya tengo mis índices de mis documentos
cambie el charset a iso-8859-1 y ya pude ver las ñ y acentos, ahora ligué el
buscador a mi aplicación y desde páginas jsp se hacen búsquedas, el problema
es que cuando el usuario escribe en el text que pide el parámetro de la
consulta debe escribir los acentos, y eso es justamente lo que no quiero,
leí un poco y creo que ocupo incluir la clase isolatin o algo así, pero no
sé bien como, primero se que la debo incluir en el config.xml o en el
schema.xml pero también debo bajar un jar o algo así de la clase isolatin? o
que debo hacer no sé... por favor alguna ayuda!!!? me estoy volviendo loca!
:(
Hello!
I too have the same problem as I have my documents indexes change charset to
iso-8859-1, and I could see ñ and accents, now linking the browser to my
application from jsp pages are searched, the problem is that when the user
types in the text that requests the parameter of the query must write
accents, and this is what we do not want to, I read a bit and I think the
deal include ISOLATINO class or something, but I do not know as well, first
is that I include in the config.xml or the schema.xml but I download a jar
or something of the kind ISOLATINO? or that I do not know ... please some
help !!!?. I'm going crazy! : (
--
View this message in context: http://www.nabble.com/Best-way-to-index-without-diacritics-tp18935599p23159812.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Best way to index without diacritics
Posted by Norberto Meijome <fr...@meijome.net>.
On Thu, 14 Aug 2008 11:34:47 -0400
"Steven A Rowe" <sa...@syr.edu> wrote:
[...]
> The kind of filter Walter is talking about - a generalized language-aware character normalization Solr/Lucene filter - does not yet exist. My guess is that if/when it does materialize, both the Solr and the Lucene projects will want to have it. Historically, most functionality shared by Solr and Lucene is eventually hosted by Lucene, since Solr has a Lucene dependency, but not vice-versa.
>
> So, yes, Solr would be responsible for hosting configuration for such a filter, but the responsibility for doing something with the configuration would be Lucene's responsibility, assuming that Lucene would (eventually) host the filter and Solr would host a factory over the filter.
>
> Steve
thanks for the thorough explanation ,Steve .
B
_________________________
{Beto|Norberto|Numard} Meijome
"Throughout the centuries there were [people] who took first steps down new paths armed only with their own vision."
Ayn Rand
I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
RE: Best way to index without diacritics
Posted by Steven A Rowe <sa...@syr.edu>.
Hi Norberto,
On 08/14/2008 at 8:10 AM, Norberto Meijome wrote:
> > On 8/13/08 9:16 AM, "Steven A Rowe" <sa...@syr.edu> wrote:
> >
> > > Hi Norberto,
> > >
> > > https://issues.apache.org/jira/browse/LUCENE-1343
>
> hi Steve,
> thanks for the pointer. this is a Lucene entry... I thought the
> Latin-filter was a SOLR feature? I, for one, definitely meant a SOLR filter.
A fair portion of Solr is a set of wrappers over Lucene functionality. ISOLatin1FilterFactory, for example, wraps Lucene's ISOLatin1AccentFilter. Here is the entirety of the Solr code:
public class ISOLatin1AccentFilterFactory extends BaseTokenFilterFactory {
public ISOLatin1AccentFilter create(TokenStream input) {
return new ISOLatin1AccentFilter(input);
}
}
Of course, BaseTokenFilterFactory brings more to the party, but my point is that adding Lucene filters to Solr is generally a trivial exercise - a Solr ...FilterFactory around LUCENE-1343 would not be much longer than the four lines listed above, since the configuration aspects are already handled by BaseTokenFilterFactory.
> Given what Walter rightly pointed out about differences in language, I suspect
> it would be a SOLR-level thing - fieldType name="textDE" language="DE" would
> apply the filter of unicode chars to {ascii?} with the appropriate mapping
> for German, etc.
>
> Or is this that Lucene would / should take care of ?
The kind of filter Walter is talking about - a generalized language-aware character normalization Solr/Lucene filter - does not yet exist. My guess is that if/when it does materialize, both the Solr and the Lucene projects will want to have it. Historically, most functionality shared by Solr and Lucene is eventually hosted by Lucene, since Solr has a Lucene dependency, but not vice-versa.
So, yes, Solr would be responsible for hosting configuration for such a filter, but the responsibility for doing something with the configuration would be Lucene's responsibility, assuming that Lucene would (eventually) host the filter and Solr would host a factory over the filter.
Steve
Re: Best way to index without diacritics
Posted by Norberto Meijome <fr...@meijome.net>.
( 2 in 1 reply)
On Wed, 13 Aug 2008 09:59:21 -0700
Walter Underwood <wu...@netflix.com> wrote:
> Stripping accents doesn't quite work. The correct translation
> is language-dependent. In German, o-dieresis should turn into
> "oe", but in English, it shoulde be "o" (as in "co__perate" or
> "M__tley Cr__e"). In Swedish, it should not be converted at all.
Hi Walter,
understood. This goes back to the question of language-specific field
definitions / parsers... more on this below.
>
> There are other character-to-string conversions: ae-ligature
> to "ae", "__" to "ss", and so on. Luckily, those are independent
> of language.
>
> wunder
>
> On 8/13/08 9:16 AM, "Steven A Rowe" <sa...@syr.edu> wrote:
>
> > Hi Norberto,
> >
> > https://issues.apache.org/jira/browse/LUCENE-1343
hi Steve,
thanks for the pointer. this is a Lucene entry... I thought the Latin-filter
was a SOLR feature? I, for one, definitely meant a SOLR filter.
Given what Walter rightly pointed out about differences in language, I suspect
it would be a SOLR-level thing - fieldType name="textDE" language="DE" would
apply the filter of unicode chars to {ascii?} with the appropriate mapping for
German, etc.
Or is this that Lucene would / should take care of ?
B
_________________________
{Beto|Norberto|Numard} Meijome
"I've dirtied my hands writing poetry, for the sake of seduction; that is, for
the sake of a useful cause." Dostoevsky
I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.
Re: Best way to index without diacritics
Posted by Walter Underwood <wu...@netflix.com>.
Stripping accents doesn't quite work. The correct translation
is language-dependent. In German, o-dieresis should turn into
"oe", but in English, it shoulde be "o" (as in "coöperate" or
"Mötley Crüe"). In Swedish, it should not be converted at all.
There are other character-to-string conversions: ae-ligature
to "ae", "ß" to "ss", and so on. Luckily, those are independent
of language.
wunder
On 8/13/08 9:16 AM, "Steven A Rowe" <sa...@syr.edu> wrote:
> Hi Norberto,
>
> https://issues.apache.org/jira/browse/LUCENE-1343
>
> :)
>
> Steve
>
> On 08/13/2008 at 12:35 AM, Norberto Meijome wrote:
>> On Tue, 12 Aug 2008 11:44:42 -0400
>> "Steven A Rowe" <sa...@syr.edu> wrote:
>>
>>> Solr is Unicode aware. The ISOLatin1AccentFilterFactory
>> handles diacritics for the ISO Latin-1 section of the Unicode
>> character set. UTF (do you mean UTF-8?) is a (set of)
>> Unicode serialization(s), and once Solr has deserialized it,
>> it is just Unicode characters (Java's in-memory UTF-16
>> representation).
>>>
>>> So as long as you're only concerned about removing
>> diacritics from the set of Unicode characters that overlaps
>> ISO Latin-1, and not about other Unicode characters, then
>> ISOLatin1AccentFilterFactory should work for you.
>>
>> hi,
>> do you know if anyone has implemented a similar filter using
>> icu and mapping (a lot more of) UTF-8 to ascii ?
>>
>> B
>>
>> _________________________
>> {Beto|Norberto|Numard} Meijome
RE: Best way to index without diacritics
Posted by Steven A Rowe <sa...@syr.edu>.
Hi Norberto,
https://issues.apache.org/jira/browse/LUCENE-1343
:)
Steve
On 08/13/2008 at 12:35 AM, Norberto Meijome wrote:
> On Tue, 12 Aug 2008 11:44:42 -0400
> "Steven A Rowe" <sa...@syr.edu> wrote:
>
> > Solr is Unicode aware. The ISOLatin1AccentFilterFactory
> handles diacritics for the ISO Latin-1 section of the Unicode
> character set. UTF (do you mean UTF-8?) is a (set of)
> Unicode serialization(s), and once Solr has deserialized it,
> it is just Unicode characters (Java's in-memory UTF-16
> representation).
> >
> > So as long as you're only concerned about removing
> diacritics from the set of Unicode characters that overlaps
> ISO Latin-1, and not about other Unicode characters, then
> ISOLatin1AccentFilterFactory should work for you.
>
> hi,
> do you know if anyone has implemented a similar filter using
> icu and mapping (a lot more of) UTF-8 to ascii ?
>
> B
>
> _________________________
> {Beto|Norberto|Numard} Meijome
>
> "He has the attention span of a lightning bolt."
> Robert Redford
>
> I speak for myself, not my employer. Contents may be hot.
> Slippery when wet. Reading disclaimers makes you go blind.
> Writing them is worse. You have been Warned.
>
Re: Best way to index without diacritics
Posted by Norberto Meijome <fr...@meijome.net>.
On Tue, 12 Aug 2008 11:44:42 -0400
"Steven A Rowe" <sa...@syr.edu> wrote:
> Solr is Unicode aware. The ISOLatin1AccentFilterFactory handles diacritics for the ISO Latin-1 section of the Unicode character set. UTF (do you mean UTF-8?) is a (set of) Unicode serialization(s), and once Solr has deserialized it, it is just Unicode characters (Java's in-memory UTF-16 representation).
>
> So as long as you're only concerned about removing diacritics from the set of Unicode characters that overlaps ISO Latin-1, and not about other Unicode characters, then ISOLatin1AccentFilterFactory should work for you.
hi,
do you know if anyone has implemented a similar filter using icu and mapping (a lot more of) UTF-8 to ascii ?
B
_________________________
{Beto|Norberto|Numard} Meijome
"He has the attention span of a lightning bolt."
Robert Redford
I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
RE: Best way to index without diacritics
Posted by Steven A Rowe <sa...@syr.edu>.
Hi Alejandro,
Solr is Unicode aware. The ISOLatin1AccentFilterFactory handles diacritics for the ISO Latin-1 section of the Unicode character set. UTF (do you mean UTF-8?) is a (set of) Unicode serialization(s), and once Solr has deserialized it, it is just Unicode characters (Java's in-memory UTF-16 representation).
So as long as you're only concerned about removing diacritics from the set of Unicode characters that overlaps ISO Latin-1, and not about other Unicode characters, then ISOLatin1AccentFilterFactory should work for you.
Steve
On 08/11/2008 at 7:22 PM, Alejandro Garza Gonzalez wrote:
> I have utf-8 content that I wat to index, however I want searches
> without diacritics to return results.
>
> For example, a document with the words "nino en mexico" should return
> results like a document with the phrase "Niño en México".
>
> Ideally, exact diacritic matches should score higher (searching for
> "niño" exactly should make a document with "niño" score higher than a
> document with "nino")
>
> Any pointers on how to do this? I found about the
> /solr/.ISOLatin1AccentFilterFactory but it seems to only strip
> diacritics from iso-latin characters. How about UTF diacritics? --
> _________________ ___ _ _ _ _ _ _ _ *Ing. Alejandro Garza González*
> Director, Tecnología e Innovación, Biblioteca Tecnológico de Monterrey,
> Campus Monterrey
>
> Tel.: 52(81) 8358-1400 ext. 4037 Fax: 52(81) 8328-4067
> Enlace Intercampus: 80 689 4037
> http://biblioteca.mty.itesm.mx
>
> El contenido de este mensaje de datos no se considera oferta, propuesta
> o acuerdo, sino hasta que sea confirmado en documento por escrito que
> contenga la firma autógrafa del apoderado legal del ITESM. El contenido
> de este mensaje de datos es confidencial y se entiende dirigido y para
> uso exclusivo del destinatario, por lo que no podrá distribuirse y/o
> difundirse por ningún medio sin la previa autorización del emisor
> original. Si usted no es el destinatario, se le prohíbe su utilización
> total o parcial para cualquier fin.
>
> The content of this data transmission must not be considered
> an offer,
> proposal, understanding or agreement unless it is confirmed in a
> document signed by a legal representative of ITESM. The
> content of this
> data transmission is confidential and is intended to be
> delivered only
> to the addressees. Therefore, it shall not be distributed and/or
> disclosed through any means without the authorization of the original
> sender. If you are not the addressee, you are forbidden from
> using it,
> either totally or partially, for any purpose.
>
>