You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Thorsten Heit <th...@gmx.de.INVALID> on 2024/08/14 13:18:09 UTC
Searching for special characters in documents
Hi,
this is the first time I'm writing to this list, so hi to all :-)
I'm having problems querying text having special characters inside (see
https://solr.apache.org/guide/solr/latest/query-guide/standard-query-parser.html#escaping-special-charaters).
My setup:
Solr 9.6.1 running as a standalone server system under Java 21 on an
internal Linux VM (Ubuntu 24.04).
For testing purposes I created a new core "test" and uploaded a few
sample documents to it:
{
"responseHeader":{
"status":0,
"QTime":0,
"params":{
"q":"*:*",
"indent":"true",
"q.op":"OR",
"useParams":"",
"_":"1723547755451"
}
},
"response":{
"numFound":7,
"start":0,
"numFoundExact":true,
"docs":[{
"id":"70",
"resourcename":"beispiel.txt",
"content_type":["text/plain; charset=windows-1252"],
"content":[" \n \n \n \n \n \n \n \n \n \n
Suchtext:\r\nab_dc\r\n \n "],
"_version_":1801822834550374400
},{
"id":"71",
"resourcename":"beispiel2.txt",
"content_type":["text/plain; charset=windows-1252"],
"content":[" \n \n \n \n \n \n \n \n \n \n
Suchtext:\r\nab-dc\r\n \n "],
"_version_":1801823062283255808
},{
"id":"72",
"resourcename":"beispiel3.txt",
"content_type":["text/plain; charset=windows-1252"],
"content":[" \n \n \n \n \n \n \n \n \n \n Dies ist ein
langer Suchtext:\r\nab-de\r\ndef+hi\r\nkl-nop\r\n \n "],
"_version_":1806915982686420992
},{
"id":"73",
"resourcename":"beispiel4.txt",
"content_type":["text/plain; charset=windows-1252"],
"content":[" \n \n \n \n \n \n \n \n \n \n
ab-cd\r\nde-fg\r\n \n "],
"_version_":1806917322172006400
},{
"id":"74",
"resourcename":"beispiel2-1.txt",
"content_type":["text/plain; charset=windows-1252"],
"content":[" \n \n \n \n \n \n \n \n \n \n Dies ist ein
langer Suchtext:\r\nabedc\r\ndef+ghi\r\n \n "],
"_version_":1807270704395059200
},{
"id":"75",
"resourcename":"beispiel2-2.txt",
"content_type":["text/plain; charset=windows-1252"],
"content":[" \n \n \n \n \n \n \n \n \n \n Dies ist ein
langer Suchtext:\r\nab-dc\r\ndefxghi\r\n \n "],
"_version_":1807270722296348672
},{
"id":"76",
"resourcename":"beispiel2-3.txt",
"content_type":["text/plain; charset=windows-1252"],
"content":[" \n \n \n \n \n \n \n \n \n \n Dies ist ein
langer Suchtext:\r\nabedc\r\ndefxghi\r\n \n "],
"_version_":1807270740219658240
}]
}
}
The problem is that I haven't found out how to correctly search for
documents with a "-" in it by using wildcards (* and ?). Some queries
seem to work while others don't...
The query itself is basically the same:
q=...&q.op=AND&fl=id,resourcename&sort=id+asc&start=0&rows=2147483647
and differs only in the value of "q".
My queries:
q: *uchtex*
=> ok, 6 documents found (#70, #71, #72, #74, #75, #76)
q: uchtex*
=> ok, 0 documents found
q: Suchtex*
=> ok, 6 documents found (#70, #71, #72, #74, #75, #76)
q: b?d
=> ok, 0 documents found
q: b?d*
=> ok, 0 documents found
q: *b-d*
=> ok, 0 documents found (because "-" isn't quoted, right?)
q: *b?d*
=> not ok, only 3 documents found: #70, #74, #76
=> missing: #71, #72, #75
q: *b*d*
=> not ok, only 3 documents found: #70, #74, #76
=> (all 7 expected)
q: ?b?d?
=> not ok, only 3 documents found: #70, #74, #76
=> missing: #71, #72, #75
q: ab*
=> ok, all 7 documents found
q: ab*d
=> not ok, 0 documents found
=> missing: #73
q: ab??d
=> not ok, 0 documents found
=> missing: #73
q: ab\-dc
=> ok, 2 documents found: #71, #75
q: ab\-d*
=> not ok, 0 documents found
=> missing: #71, #72, #75
q: ab?d*
=> not ok, 3 documents found: #70, #74, #76
=> missing: #71, #72, #75
q: *b\-d*
=> not ok, 0 documents found
=> missing: #71, #72, #75
q: *b\\-d*
=> 0
Can someone enlighten me what I'm doing wrong? Am I missing something?
Or do I misunderstand something?
Regards
Thorsten
Re: Searching for special characters in documents
Posted by Thomas Corthals <th...@klascement.net>.
Hi Thorsten,
For the sake of completeness I'm going to explain why I suggested wt=xml
but it's not actually relevant in your case.
> Adding "wt=xml" to the query params above didn't change much...
It won't change the query results at all. I just mentioned it because it
avoids confusion about the actual query Solr gets from your request.
curl -s http://localhost:8983/solr/techproducts/select?q=b\\-d
If you do that in a shell like bash, the actual query that gets sent is b\-d.
If you're not familiar with JSON, you could mistakenly assume it was b\\-d when
looking at the raw output because JSON introduces a backslash as an escape
where the shell had stripped one.
{
"responseHeader":{
"status":0,
"QTime":6,
"params":{
"q":"b\\-d"
}
},
"response":{
"numFound":0,
"start":0,
"numFoundExact":true,
"docs":[ ]
}
}
With wt=xml there is no potential for confusion when your query contains
backslashes.
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">0</int>
<lst name="params">
<str name="q">b\-d</str>
<str name="wt">xml</str>
</lst>
</lst>
<result name="response" numFound="0" start="0" numFoundExact="true">
</result>
</response>
All of this is just to make sure that you're definitely debugging the
"right" query.
> Do you have idea what might causing this?
It has nothing to do with escaping special characters. It's the tokenizer.
It splits the content field into separate tokens when indexing and
searching. It considers a hyphen as a separator between tokens just like
whitespace, so both "ab-dc" and "ab dc" are indexed as ["ab", "dc"].
However, "ab_dc" is retained as a single token. And those tokens are what
you're actually matching against. So even if you successfully escape a
hyphen, there is no token in your index that will match that literal hyphen.
Try these three values for q=... to see what that means:
- ab
- dc
- ab_dc
If you based your schema on the schemaless example that comes with Solr,
there should be an untokenized copy of the content field indexed under
content_str. Try adding df=content_str to search in this field and if it
exists you should get the expected results for your wildcard queries. (But
not for those without wildcards.)
Thomas
Op wo 28 aug 2024 om 11:05 schreef Thorsten Heit <th...@gmx.de.invalid>:
> Hi Thomas,
>
> > How exactly are you executing the queries? If you're using shell
> commands,
> > keep in mind that the shell will apply its own escaping rules to your
> > command parameters first so if you're not careful a backslash might
> already
> > be "eaten" before the actual request is fired of. Same goes for any
> string
> > escaping rules in the programming language you might be implementing your
> > test script in.
>
> I'm executing the queries from within Java by using solr-solrj-9.6.1.
> This is an excerpt of the code in question that is being used:
>
>
> var query = new SolrQuery(searchString)
> setParam("q.op", SearchMode.OR == request.getSearchMode() ? "OR" :
> "AND")
> .setFields("id", "filename")
> .setSort("id", ORDER.asc)
> .setStart(request.getStart())
> .setRows(request.getRows());
>
> final QueryResponse queryResponse;
> try {
> getReadLock().lock();
> var solrClient = getSolrClient();
> queryResponse = solrClient.query(query, METHOD.POST);
> } catch (SolrException ex) {
> throw new SolrServerException(ex);
> } finally {
> getReadLock().unlock();
> }
>
>
> I have tested the above snippet with the query strings from my initial
> post, i.e. quoted text, non-quoted text, hyphens escaped/non-escaped
> etc., but the results didn't match what I was expecting...
>
>
> Adding "wt=xml" to the query params above didn't change much...
>
> Do you have idea what might causing this?
>
>
> Regards
>
> Thorsten
>
Re: Searching for special characters in documents
Posted by Thorsten Heit <th...@gmx.de.INVALID>.
Hi Thomas,
> How exactly are you executing the queries? If you're using shell commands,
> keep in mind that the shell will apply its own escaping rules to your
> command parameters first so if you're not careful a backslash might already
> be "eaten" before the actual request is fired of. Same goes for any string
> escaping rules in the programming language you might be implementing your
> test script in.
I'm executing the queries from within Java by using solr-solrj-9.6.1.
This is an excerpt of the code in question that is being used:
var query = new SolrQuery(searchString)
setParam("q.op", SearchMode.OR == request.getSearchMode() ? "OR" :
"AND")
.setFields("id", "filename")
.setSort("id", ORDER.asc)
.setStart(request.getStart())
.setRows(request.getRows());
final QueryResponse queryResponse;
try {
getReadLock().lock();
var solrClient = getSolrClient();
queryResponse = solrClient.query(query, METHOD.POST);
} catch (SolrException ex) {
throw new SolrServerException(ex);
} finally {
getReadLock().unlock();
}
I have tested the above snippet with the query strings from my initial
post, i.e. quoted text, non-quoted text, hyphens escaped/non-escaped
etc., but the results didn't match what I was expecting...
Adding "wt=xml" to the query params above didn't change much...
Do you have idea what might causing this?
Regards
Thorsten
Re: Searching for special characters in documents
Posted by Thomas Corthals <th...@klascement.net>.
Hi Thorsten,
How exactly are you executing the queries? If you're using shell commands,
keep in mind that the shell will apply its own escaping rules to your
command parameters first so if you're not careful a backslash might already
be "eaten" before the actual request is fired of. Same goes for any string
escaping rules in the programming language you might be implementing your
test script in.
With echoParams=explicit you can see what ended up being sent to Solr, but
keep in mind that JSON also uses backslash as an escape character. This is
one case where wt=xml can actually help avoid confusion.
Thomas
Op wo 14 aug 2024 15:18 schreef Thorsten Heit <th...@gmx.de.invalid>:
> Hi,
>
> this is the first time I'm writing to this list, so hi to all :-)
>
> I'm having problems querying text having special characters inside (see
>
> https://solr.apache.org/guide/solr/latest/query-guide/standard-query-parser.html#escaping-special-charaters
> ).
>
> My setup:
> Solr 9.6.1 running as a standalone server system under Java 21 on an
> internal Linux VM (Ubuntu 24.04).
>
> For testing purposes I created a new core "test" and uploaded a few
> sample documents to it:
>
>
> {
> "responseHeader":{
> "status":0,
> "QTime":0,
> "params":{
> "q":"*:*",
> "indent":"true",
> "q.op":"OR",
> "useParams":"",
> "_":"1723547755451"
> }
> },
> "response":{
> "numFound":7,
> "start":0,
> "numFoundExact":true,
> "docs":[{
> "id":"70",
> "resourcename":"beispiel.txt",
> "content_type":["text/plain; charset=windows-1252"],
> "content":[" \n \n \n \n \n \n \n \n \n \n
> Suchtext:\r\nab_dc\r\n \n "],
> "_version_":1801822834550374400
> },{
> "id":"71",
> "resourcename":"beispiel2.txt",
> "content_type":["text/plain; charset=windows-1252"],
> "content":[" \n \n \n \n \n \n \n \n \n \n
> Suchtext:\r\nab-dc\r\n \n "],
> "_version_":1801823062283255808
> },{
> "id":"72",
> "resourcename":"beispiel3.txt",
> "content_type":["text/plain; charset=windows-1252"],
> "content":[" \n \n \n \n \n \n \n \n \n \n Dies ist ein
> langer Suchtext:\r\nab-de\r\ndef+hi\r\nkl-nop\r\n \n "],
> "_version_":1806915982686420992
> },{
> "id":"73",
> "resourcename":"beispiel4.txt",
> "content_type":["text/plain; charset=windows-1252"],
> "content":[" \n \n \n \n \n \n \n \n \n \n
> ab-cd\r\nde-fg\r\n \n "],
> "_version_":1806917322172006400
> },{
> "id":"74",
> "resourcename":"beispiel2-1.txt",
> "content_type":["text/plain; charset=windows-1252"],
> "content":[" \n \n \n \n \n \n \n \n \n \n Dies ist ein
> langer Suchtext:\r\nabedc\r\ndef+ghi\r\n \n "],
> "_version_":1807270704395059200
> },{
> "id":"75",
> "resourcename":"beispiel2-2.txt",
> "content_type":["text/plain; charset=windows-1252"],
> "content":[" \n \n \n \n \n \n \n \n \n \n Dies ist ein
> langer Suchtext:\r\nab-dc\r\ndefxghi\r\n \n "],
> "_version_":1807270722296348672
> },{
> "id":"76",
> "resourcename":"beispiel2-3.txt",
> "content_type":["text/plain; charset=windows-1252"],
> "content":[" \n \n \n \n \n \n \n \n \n \n Dies ist ein
> langer Suchtext:\r\nabedc\r\ndefxghi\r\n \n "],
> "_version_":1807270740219658240
> }]
> }
> }
>
>
> The problem is that I haven't found out how to correctly search for
> documents with a "-" in it by using wildcards (* and ?). Some queries
> seem to work while others don't...
>
> The query itself is basically the same:
>
> q=...&q.op=AND&fl=id,resourcename&sort=id+asc&start=0&rows=2147483647
>
> and differs only in the value of "q".
>
> My queries:
>
> q: *uchtex*
> => ok, 6 documents found (#70, #71, #72, #74, #75, #76)
>
> q: uchtex*
> => ok, 0 documents found
>
> q: Suchtex*
> => ok, 6 documents found (#70, #71, #72, #74, #75, #76)
>
> q: b?d
> => ok, 0 documents found
>
> q: b?d*
> => ok, 0 documents found
>
> q: *b-d*
> => ok, 0 documents found (because "-" isn't quoted, right?)
>
> q: *b?d*
> => not ok, only 3 documents found: #70, #74, #76
> => missing: #71, #72, #75
>
> q: *b*d*
> => not ok, only 3 documents found: #70, #74, #76
> => (all 7 expected)
>
> q: ?b?d?
> => not ok, only 3 documents found: #70, #74, #76
> => missing: #71, #72, #75
>
> q: ab*
> => ok, all 7 documents found
>
> q: ab*d
> => not ok, 0 documents found
> => missing: #73
>
> q: ab??d
> => not ok, 0 documents found
> => missing: #73
>
> q: ab\-dc
> => ok, 2 documents found: #71, #75
>
> q: ab\-d*
> => not ok, 0 documents found
> => missing: #71, #72, #75
>
> q: ab?d*
> => not ok, 3 documents found: #70, #74, #76
> => missing: #71, #72, #75
>
> q: *b\-d*
> => not ok, 0 documents found
> => missing: #71, #72, #75
>
> q: *b\\-d*
> => 0
>
>
> Can someone enlighten me what I'm doing wrong? Am I missing something?
> Or do I misunderstand something?
>
>
> Regards
>
> Thorsten
>
>
>