You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by in...@joinso.cat on 2024/05/14 08:49:27 UTC

Synonyms and text search

Hi!
I am using Apache Solr 5.3.1, and I setup synonyms text file.
It works as expected.

For example, in my synonyms.txt file I have:

ML,mL,Ml,ml,Machine Learning

AI,Ai,aI,ai,Artificial Intelligence


I have 2 categories in a field with this terms: Machine Learning and
Artificial Intelligence
I tagged 4 documents with Artificial Intelligence and one document without
this tag.
But in this document I added in the description the word “AI” several times.

When I search for AI it returns the 4 documents but not the one not tagged
that it has the word AI inside the description.
There is anyway to return all the document that match the synonyms and also
that the text is in inside the document?

 

Atentamente / Atentament  / Regards,
JOINSO

  _____  




IT Consulting and Advice

93 707 90 70 |  <http://www.joinso.cat/> www.joinso.cat

 

 


Nota Legal: Este correo se rige por la política de privacidad descrita
<http://www.joinso.cat/politica-de-privacidad> aquí.

Nota Legal: Aquest correu es regeix per la política de privacitat descrita
<http://www.joinso.cat/ca/politica-de-privacitat> aquí.

Legal Disclaimer: This email is governed by privacy policy described
<http://www.joinso.cat/en/privacy-policy> here.

 


RE: Synonyms and text search

Posted by in...@joinso.cat.
The schema is like this:

Field: sm_vid_Category

Field-Type:org.apache.solr.schema.StrField

 

In this field, we have this value : Artificial Intelligence

And there is 4 documents that has this value.

The description field belongs to: 

 


Field: content


Field-Type:

org.apache.solr.schema.TextField

PI Gap:

100

Docs:

 <http://admin:$so23os32@apache-solr-dev.food4rhino.com:8983/solr/#/drupal_dev/query?q=content:[*%20TO%20*]> 171

 

 

Here we can found for example, the word “AI” than in synonyms belongs to Artificial Intelligence

If I do the search with AI only returns the 4 documents that belongs to sm_vid_Category

 

But in the other document that contents this in the description:

 

AI AI AI AI AI AI AI AI
skjdskjdkjdskdj
AI AI AI AI AI AI AI AI
sdlsldsjdsjdjs
AI AI AI AI AI AI AI AI
ssdjashdkjshd asdhashdakhd

 

it never return in the search if I search “AI”, but it return the document if I search “skjdskjdkjdskdj” or “sdlsldsjdsjdjs”

 

Regards,

Jordi
JOINSO

 

-----Mensaje original-----
De: Mikhail Khludnev <mk...@apache.org> 
Enviado el: martes, 14 de mayo de 2024 15:01
Para: users@solr.apache.org
Asunto: Re: Synonyms and text search

 

It depends on schema and query parser details. You may use Analysis screen, debugQuery=true, and explainOther=id:123 to troubleshoot this case.

 

On Tue, May 14, 2024 at 11:50 AM < <ma...@joinso.cat> info@joinso.cat> wrote:

 

> Hi!

> I am using Apache Solr 5.3.1, and I setup synonyms text file.

> It works as expected.

> 

> For example, in my synonyms.txt file I have:

> 

> ML,mL,Ml,ml,Machine Learning

> 

> AI,Ai,aI,ai,Artificial Intelligence

> 

> 

> I have 2 categories in a field with this terms: Machine Learning and 

> Artificial Intelligence I tagged 4 documents with Artificial 

> Intelligence and one document without this tag.

> But in this document I added in the description the word “AI” several 

> times.

> 

> When I search for AI it returns the 4 documents but not the one not 

> tagged that it has the word AI inside the description.

> There is anyway to return all the document that match the synonyms and 

> also that the text is in inside the document?

> 

> 

> 

> Atentamente / Atentament  / Regards,

> JOINSO

> ------------------------------

> 

> [image: JOINSO IT CONSULTING AND ADVICE]

> 

> IT Consulting and Advice

> 

> 93 707 90 70 |  <http://www.joinso.cat> www.joinso.cat

> 

> 

> 

> 

> 

> 

> Nota Legal: Este correo se rige por la política de privacidad descrita 

> aquí < <http://www.joinso.cat/politica-de-privacidad> http://www.joinso.cat/politica-de-privacidad>.

> 

> Nota Legal: Aquest correu es regeix per la política de privacitat 

> descrita aquí < <http://www.joinso.cat/ca/politica-de-privacitat> http://www.joinso.cat/ca/politica-de-privacitat>.

> 

> Legal Disclaimer: This email is governed by privacy policy described 

> here < <http://www.joinso.cat/en/privacy-policy> http://www.joinso.cat/en/privacy-policy>.

> 

> 

> 

 

 

--

Sincerely yours

Mikhail Khludnev


RE: Synonyms and text search

Posted by in...@joinso.cat.
Hi!
Now my synonyms.txt file is like this:
ml,Machine Learning
ai,Artificial Intelligence
But when I search I only get results for documents categorized with " Artificial Intelligence" when  I search "ai".
But I do not get in the results list a document with the word "ai" inside.

Any idea to get all the documents categorized as " Artificial Intelligence" and also the ones with the word "ai" inside?

Regards,
Jordi
-----Mensaje original-----
De: Charlie Hull <ch...@opensourceconnections.com> 
Enviado el: miércoles, 15 de mayo de 2024 14:54
Para: users@solr.apache.org
Asunto: Re: Synonyms and text search

oh and you generally use synonyms for the 'AI = artificial intelligence' 
use case, the lower/upper case stuff should be handled by the analysis chain (that turns your source documents into a normalised form for
indexing)
https://solr.apache.org/guide/solr/latest/indexing-guide/document-analysis.html

Charlie

On 14/05/2024 14:01, Mikhail Khludnev wrote:
> It depends on schema and query parser details. You may use Analysis 
> screen, debugQuery=true, and explainOther=id:123 to troubleshoot this case.
>
> On Tue, May 14, 2024 at 11:50 AM <in...@joinso.cat> wrote:
>
>> Hi!
>> I am using Apache Solr 5.3.1, and I setup synonyms text file.
>> It works as expected.
>>
>> For example, in my synonyms.txt file I have:
>>
>> ML,mL,Ml,ml,Machine Learning
>>
>> AI,Ai,aI,ai,Artificial Intelligence
>>
>>
>> I have 2 categories in a field with this terms: Machine Learning and 
>> Artificial Intelligence I tagged 4 documents with Artificial 
>> Intelligence and one document without this tag.
>> But in this document I added in the description the word “AI” several 
>> times.
>>
>> When I search for AI it returns the 4 documents but not the one not 
>> tagged that it has the word AI inside the description.
>> There is anyway to return all the document that match the synonyms 
>> and also that the text is in inside the document?
>>
>>
>>
>> Atentamente / Atentament  / Regards,
>> JOINSO
>> ------------------------------
>>
>> [image: JOINSO IT CONSULTING AND ADVICE]
>>
>> IT Consulting and Advice
>>
>> 93 707 90 70 | www.joinso.cat
>>
>>
>>
>>
>>
>>
>> Nota Legal: Este correo se rige por la política de privacidad 
>> descrita aquí <http://www.joinso.cat/politica-de-privacidad>.
>>
>> Nota Legal: Aquest correu es regeix per la política de privacitat 
>> descrita aquí <http://www.joinso.cat/ca/politica-de-privacitat>.
>>
>> Legal Disclaimer: This email is governed by privacy policy described 
>> here <http://www.joinso.cat/en/privacy-policy>.
>>
>>
>>
>
--
Charlie Hull - Managing Consultant at OpenSource Connections Limited Founding member of The Search Network and co-author of Searching the Enterprise
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828

OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin Amtsgericht Charlottenburg | HRB 230712 B
Geschäftsführer: John M. Woodell | David E. Pugh
Finanzamt: Berlin Finanzamt für Körperschaften II



Re: Synonyms and text search

Posted by Charlie Hull <ch...@opensourceconnections.com>.
oh and you generally use synonyms for the 'AI = artificial intelligence' 
use case, the lower/upper case stuff should be handled by the analysis 
chain (that turns your source documents into a normalised form for 
indexing)
https://solr.apache.org/guide/solr/latest/indexing-guide/document-analysis.html

Charlie

On 14/05/2024 14:01, Mikhail Khludnev wrote:
> It depends on schema and query parser details. You may use Analysis screen,
> debugQuery=true, and explainOther=id:123 to troubleshoot this case.
>
> On Tue, May 14, 2024 at 11:50 AM <in...@joinso.cat> wrote:
>
>> Hi!
>> I am using Apache Solr 5.3.1, and I setup synonyms text file.
>> It works as expected.
>>
>> For example, in my synonyms.txt file I have:
>>
>> ML,mL,Ml,ml,Machine Learning
>>
>> AI,Ai,aI,ai,Artificial Intelligence
>>
>>
>> I have 2 categories in a field with this terms: Machine Learning and
>> Artificial Intelligence
>> I tagged 4 documents with Artificial Intelligence and one document without
>> this tag.
>> But in this document I added in the description the word “AI” several
>> times.
>>
>> When I search for AI it returns the 4 documents but not the one not tagged
>> that it has the word AI inside the description.
>> There is anyway to return all the document that match the synonyms and
>> also that the text is in inside the document?
>>
>>
>>
>> Atentamente / Atentament  / Regards,
>> JOINSO
>> ------------------------------
>>
>> [image: JOINSO IT CONSULTING AND ADVICE]
>>
>> IT Consulting and Advice
>>
>> 93 707 90 70 | www.joinso.cat
>>
>>
>>
>>
>>
>>
>> Nota Legal: Este correo se rige por la política de privacidad descrita
>> aquí <http://www.joinso.cat/politica-de-privacidad>.
>>
>> Nota Legal: Aquest correu es regeix per la política de privacitat descrita
>> aquí <http://www.joinso.cat/ca/politica-de-privacitat>.
>>
>> Legal Disclaimer: This email is governed by privacy policy described here
>> <http://www.joinso.cat/en/privacy-policy>.
>>
>>
>>
>
-- 
Charlie Hull - Managing Consultant at OpenSource Connections Limited
Founding member of The Search Network and co-author of Searching the Enterprise
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828

OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
Amtsgericht Charlottenburg | HRB 230712 B
Geschäftsführer: John M. Woodell | David E. Pugh
Finanzamt: Berlin Finanzamt für Körperschaften II


Re: Synonyms and text search

Posted by Mikhail Khludnev <mk...@apache.org>.
It depends on schema and query parser details. You may use Analysis screen,
debugQuery=true, and explainOther=id:123 to troubleshoot this case.

On Tue, May 14, 2024 at 11:50 AM <in...@joinso.cat> wrote:

> Hi!
> I am using Apache Solr 5.3.1, and I setup synonyms text file.
> It works as expected.
>
> For example, in my synonyms.txt file I have:
>
> ML,mL,Ml,ml,Machine Learning
>
> AI,Ai,aI,ai,Artificial Intelligence
>
>
> I have 2 categories in a field with this terms: Machine Learning and
> Artificial Intelligence
> I tagged 4 documents with Artificial Intelligence and one document without
> this tag.
> But in this document I added in the description the word “AI” several
> times.
>
> When I search for AI it returns the 4 documents but not the one not tagged
> that it has the word AI inside the description.
> There is anyway to return all the document that match the synonyms and
> also that the text is in inside the document?
>
>
>
> Atentamente / Atentament  / Regards,
> JOINSO
> ------------------------------
>
> [image: JOINSO IT CONSULTING AND ADVICE]
>
> IT Consulting and Advice
>
> 93 707 90 70 | www.joinso.cat
>
>
>
>
>
>
> Nota Legal: Este correo se rige por la política de privacidad descrita
> aquí <http://www.joinso.cat/politica-de-privacidad>.
>
> Nota Legal: Aquest correu es regeix per la política de privacitat descrita
> aquí <http://www.joinso.cat/ca/politica-de-privacitat>.
>
> Legal Disclaimer: This email is governed by privacy policy described here
> <http://www.joinso.cat/en/privacy-policy>.
>
>
>


-- 
Sincerely yours
Mikhail Khludnev