You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mike Drob <md...@apache.org> on 2020/05/01 15:09:03 UTC
Re: Possible issue with Stemming and nouns ended with suffix 'ion'
Jhonny,
Are you planning on reporting the issue to snowball, or would you prefer
one of us take care of it?
If you do report it, please share the link to the issue or mail archive
back here so that we know when it is resolved and can update our
dependencies.
Thanks,
Mike
On Thu, Apr 30, 2020 at 5:37 PM Jhonny Lopez <jh...@publicismedia.com>
wrote:
> Yes, sounds like worth it.
>
> Thanks guys!
>
> -----Original Message-----
> From: Mike Drob <md...@apache.org>
> Sent: jueves, 30 de abril de 2020 5:30 p. m.
> To: solr-user@lucene.apache.org
> Subject: Re: Possible issue with Stemming and nouns ended with suffix 'ion'
>
> This email has been sent from a source external to Publicis Groupe. Please
> use caution when clicking links or opening attachments.
> Cet email a été envoyé depuis une source externe à Publicis Groupe.
> Veuillez faire preuve de prudence lorsque vous cliquez sur des liens ou
> lorsque vous ouvrez des pièces jointes.
>
>
>
> Is this worth filing a bug/suggestion to the folks over at
> snowballstem.org?
>
> On Thu, Apr 30, 2020 at 4:08 PM Audrey Lorberfeld -
> Audrey.Lorberfeld@ibm.com <Au...@ibm.com> wrote:
>
> > I agree with Erick. I think that's just how the cookie crumbles when
> > stemming. If you have some time on your hands, you can integrate
> > OpenNLP with your Solr instance and start using the lemmas of tokens
> > instead of the stems. In this case, I believe if you were to lemmatize
> > both "identify" and "identification," they would both condense to
> "identify."
> >
> > Best,
> > Audrey
> >
> > On 4/30/20, 3:54 PM, "Erick Erickson" <er...@gmail.com> wrote:
> >
> > They are being stemmed to two different tokens, “identif” and
> > “identifi”. Stemming is algorithmic and imperfect and in this case
> > you’re getting bitten by that algorithm. It looks like you’re using
> > PorterStemFilter, if you want you can look up the exact algorithm, but
> > I don’t think it’s a bug, just one of those little joys of English...
> >
> > To get a clearer picture of exactly what’s being searched, try
> > adding &debug=query to your query, in particular looking at the parsed
> > query that’s returned. That’ll tell you a bunch. In this particular
> > case I don’t think it’ll tell you anything more, but for future…
> >
> > Best,
> > Erick
> >
> > On, and un-checking the ‘verbose’ box on the analysis page removes
> > a lot of distraction, the detailed information is often TMI ;)
> >
> > > On Apr 30, 2020, at 2:51 PM, Jhonny Lopez <
> > jhonny.lopez@publicismedia.com> wrote:
> > >
> > > Sure, rewriting the message with links for images:
> > >
> > >
> > > We’re facing an issue with stemming in solr. Most of the cases
> > are working correctly, for example, if we search for bidding, solr
> > brings results for bidding, bid, bids, etc. However, with nouns ended
> with ‘ion’
> > suffix, stemming is not working. Even when analyzers seems to have
> > correct stemming of the word, the results are not reflecting that. One
> > example. If I search ‘identifying’, this is the output:
> > >
> > > Analyzer (image link):
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__1drv.ms_u_s-21AlRTlFq8tQbShd4-2DCp40Cmc0QioS0A-3Fe-3D1f3GJp&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=_8ViuZIeSRdQjONA8yHWPZIBlhj291HU3JpNIx5a55M&m=8Xt1N2A4ODj--DlLb242c8JMnJr6nWQIwcKjiDiA__s&s=U-Wmu118X5bfNxDnADO_6ompf9kUxZYHj1DZM2lG4jo&e=
> > >
> > > A clip of results:
> > > "haschildren_b":false,
> > > "isbucket_text_s":"0",
> > > "sectionbody_t":"\n\n\nIn order to identify 1st price
> > auctions, leverage the proprietary tools available or manually pull a
> > log file report to understand the trends and gauge auction spread
> > overtime to assess the impact of variable auction
> dynamics.\n\n\n\n\n\n\n",
> > > "parsedupdatedby_s":"sitecorecarvaini",
> > > "sectionbody_t_en":"\n\n\nIn order to identify 1st price
> > auctions, leverage the proprietary tools available or manually pull a
> > log file report to understand the trends and gauge auction spread
> > overtime to assess the impact of variable auction
> dynamics.\n\n\n\n\n\n\n",
> > > "hide_section_b":false
> > >
> > >
> > > As you can see, it has used the stemming correctly and brings
> > results for other words based in the root, in this case “Identify”.
> > >
> > > However, if I search for “Identification”, this is the output:
> > >
> > > Analyzer (imagelink):
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__1drv.ms_u_s-21AlRTlFq8tQbShd49RpiQObzMgSjVhA&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=_8ViuZIeSRdQjONA8yHWPZIBlhj291HU3JpNIx5a55M&m=8Xt1N2A4ODj--DlLb242c8JMnJr6nWQIwcKjiDiA__s&s=5RlkLH-90sYc4nyIgnPO9MsBlyh7iWSOphEVdjUvTIE&e=
> > >
> > >
> > > Even with proper stemming, solr is only bringing results for the
> > word identification (or identifications) but nothing else.
> > >
> > > The queries are over the same field that has the Porter Stemming
> > Filter applied for both, query and index. This behavior is consistent
> > with other ‘ion’ ended nouns: representation, modification, etc.
> > >
> > > Solr Version: 8.1. Does anyone know why is it happening? Is it a
> bug?
> > >
> > > Thanks.
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > >
> > > From: Erick Erickson <er...@gmail.com>
> > >
> > > Sent: jueves, 30 de abril de 2020 1:47 p. m.
> > >
> > > To: solr-user@lucene.apache.org
> > >
> > > Subject: Re: Possible issue with Stemming and nouns ended with
> > suffix 'ion'
> > >
> > >
> > >
> > > This email has been sent from a source external to Publicis Groupe.
> > Please use caution when clicking links or opening attachments.
> > >
> > > Cet email a été envoyé depuis une source externe à Publicis Groupe.
> > Veuillez faire preuve de prudence lorsque vous cliquez sur des liens
> > ou lorsque vous ouvrez des pièces jointes.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > The mail server is pretty aggressive about stripping links, so
> > we can’t see the images.
> > >
> > >
> > >
> > > Could you put them somewhere and paste a link?
> > >
> > >
> > >
> > > Best,
> > >
> > > Erick
> > >
> > >
> > >
> > >> On Apr 30, 2020, at 2:40 PM, Jhonny Lopez <
> > jhonny.lopez@publicismedia.com> wrote:
> > >
> > >>
> > >
> > >> We’re facing an issue with stemming in solr. Most of the cases
> > are working correctly, for example, if we search for bidding, solr
> > brings results for bidding, bid, bids, etc. However, with nouns ended
> with ‘ion’
> > suffix, stemming is not working. Even when analyzers seems to have
> > correct stemming of the word, the results are not reflecting that. One
> > example. If I search ‘identifying’, this is the output:
> > >
> > >>
> > >
> > >> Analyzer (image):
> > >
> > >>
> > >
> > >> A clip of results:
> > >
> > >> "haschildren_b":false,
> > >
> > >> "isbucket_text_s":"0",
> > >
> > >> "sectionbody_t":"\n\n\nIn order to identify 1st price
> > auctions, leverage the proprietary tools available or manually pull a
> > log file report to understand the trends and gauge auction spread
> > overtime to assess the impact of variable auction
> dynamics.\n\n\n\n\n\n\n",
> > >
> > >> "parsedupdatedby_s":"sitecorecarvaini",
> > >
> > >> "sectionbody_t_en":"\n\n\nIn order to identify 1st price
> > auctions, leverage the proprietary tools available or manually pull a
> > log file report to understand the trends and gauge auction spread
> > overtime to assess the impact of variable auction
> dynamics.\n\n\n\n\n\n\n",
> > >
> > >> "hide_section_b":false
> > >
> > >>
> > >
> > >>
> > >
> > >> As you can see, it has used the stemming correctly and brings
> > results for other words based in the root, in this case “Identify”.
> > >
> > >>
> > >
> > >> However, if I search for “Identification”, this is the output:
> > >
> > >>
> > >
> > >> Analyzer (image):
> > >
> > >>
> > >
> > >> Even with proper stemming, solr is only bringing results for
> > the word identification (or identifications) but nothing else.
> > >
> > >>
> > >
> > >> The queries are over the same field that has the Porter
> > Stemming Filter applied for both, query and index. This behavior is
> > consistent with other ‘ion’ ended nouns: representation, modification,
> etc.
> > >
> > >>
> > >
> > >> Solr Version: 8.1. Does anyone know why is it happening? Is it
> > a bug?
> > >
> > >>
> > >
> > >> Thanks.
> > >
> > >>
> > >
> > >>
> > >
> > >>
> > >
> > >>
> > >
> > >> Jhonny Lopez
> > >
> > >> Technical Architect
> > >
> > >> Avenida Calle 26 No. 92 - 32, Edificio BTS3
> > >
> > >> APDO. 128-1255 Bogota
> > >
> > >> T: +573006805461
> > >
> > >> jhonny.lopez@publicismedia.com
> > >
> > >> www.prodigious.com
> > >
> > >>
> > >
> > >>
> > >
> > >>
> > >
> > >>
> > >
> > >>
> > >
> > >>
> > >
> > >>
> > ----------------------------------------------------------------------
> > >
> > >> -- Disclaimer The information in this email and any attachments
> may
> > >
> > >> contain proprietary and confidential information that is
> > intended for the addressee(s) only. If you are not the intended
> > recipient, you are hereby notified that any disclosure, copying,
> > distribution, retention or use of the contents of this information is
> > prohibited. When addressed to our clients or vendors, any information
> > contained in this e-mail or any attachments is subject to the terms
> > and conditions in any governing contract. If you have received this
> > e-mail in error, please immediately contact the sender and delete the
> e-mail.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > ------------------------------------------------------------------------
> > > Disclaimer The information in this email and any attachments may
> > contain proprietary and confidential information that is intended for
> > the
> > addressee(s) only. If you are not the intended recipient, you are
> > hereby notified that any disclosure, copying, distribution, retention
> > or use of the contents of this information is prohibited. When
> > addressed to our clients or vendors, any information contained in this
> > e-mail or any attachments is subject to the terms and conditions in
> > any governing contract. If you have received this e-mail in error,
> > please immediately contact the sender and delete the e-mail.
> >
> >
> >
> >
>
>
>
>
> ------------------------------------------------------------------------
> Disclaimer The information in this email and any attachments may contain
> proprietary and confidential information that is intended for the
> addressee(s) only. If you are not the intended recipient, you are hereby
> notified that any disclosure, copying, distribution, retention or use of
> the contents of this information is prohibited. When addressed to our
> clients or vendors, any information contained in this e-mail or any
> attachments is subject to the terms and conditions in any governing
> contract. If you have received this e-mail in error, please immediately
> contact the sender and delete the e-mail.
>