You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Dmitri Maziuk <dm...@gmail.com> on 2024/11/05 21:44:41 UTC
configset api in 8.11?
Hi all,
anyone knows if
```
curl -X POST -H 'Content-type:application/octet-stream" \
--data-binary @solrconfig.xml
"http://solrnode:8983/solr/admin/configs?action=UPLOAD&name=myindex&filePath=solrconfig.xml&overwrite=true"
```
is supposed to work in 8.11?
I am getting a 500 from `org.noggit.JSONParser$ParseException: JSON
Parse Error` which is sort of understandable as none of that is JSON...
but what does JSON parser have to do with anything?
TIA,
Dima
Re: configset api in 8.11?
Posted by Dmitri Maziuk <dm...@gmail.com>.
On 11/7/24 14:12, Jan Høydahl wrote:
> Try -cmd put to insert either an empty string or {} and it should start working. No idea how the null got there…
>
Thank you Jan, that worked.
Dima
Re: configset api in 8.11?
Posted by Jan Høydahl <ja...@cominvent.com>.
Try -cmd put to insert either an empty string or {} and it should start working. No idea how the null got there…
Jan Høydahl
> On 7 Nov 2024, at 19:32, Dmitri Maziuk <dm...@gmail.com> wrote:
>
> PS. can that NUL be coming from one of the config files? -- I'm thinking BOM, thankyouverymuch dear Miscrosoft?
>
> Dima
>
Re: configset api in 8.11?
Posted by Dmitri Maziuk <dm...@gmail.com>.
PS. can that NUL be coming from one of the config files? -- I'm thinking
BOM, thankyouverymuch dear Miscrosoft?
Dima
Re: configset api in 8.11?
Posted by "Rieger, Peter" <pe...@esemos.de>.
Hi,
you used the Solr-Port to send stuff with zkcli, the "zk" is to the best of my knowledge an abbreviation for zookeeper so we use eg:
/opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:2181 -cmd upconfig -confdir conf -confname <very privat conf name>
I hope it helps, but I'm honestly too lazy to read the whole thread. We work with Solr since clouds were just nice on a blue sky (more than 20 years for our customers and the API is not stable but the zkcli.sh script is. We use that for ages (and many releases).
Best
Peter
PS: we tried the API but the time invest is to high imho. Who pays that, when something working is part of the same release drop?
Am 07.11.24, 19:24 schrieb "Dmitri Maziuk" <dmitri.maziuk@gmail.com <ma...@gmail.com>>:
On 11/7/24 11:51, Jan Høydahl wrote:
> /opt/solr/server/scripts/cloud-scripts/zkcli.sh -z localhost:9983 -cmd get /configs/foo >foo.txt
>
> It will write the content to "foo.txt", and according to the stack trace, that file will be non-empty and contain non-JSON.
Indeed, it contains a single NULL byte.
Attachments are stripped off but in the UI 2 out of my 4 indexes have
"not utf8 Content" and the other 2 don't -- I never noticed that until
now. The above returns a single 0x00 for both of those. Curioser and
curioser.
Any ideas how to fix it?
Thanks again,
Dima
Re: configset api in 8.11?
Posted by Dmitri Maziuk <dm...@gmail.com>.
On 11/7/24 11:51, Jan Høydahl wrote:
> /opt/solr/server/scripts/cloud-scripts/zkcli.sh -z localhost:9983 -cmd get /configs/foo >foo.txt
>
> It will write the content to "foo.txt", and according to the stack trace, that file will be non-empty and contain non-JSON.
Indeed, it contains a single NULL byte.
Attachments are stripped off but in the UI 2 out of my 4 indexes have
"not utf8 Content" and the other 2 don't -- I never noticed that until
now. The above returns a single 0x00 for both of those. Curioser and
curioser.
Any ideas how to fix it?
Thanks again,
Dima
Re: configset api in 8.11?
Posted by Jan Høydahl <ja...@cominvent.com>.
In my screenshot (not sure it made it to the list) you see I selected my configset folder and it says "Node configs/foo has no utf8 content".
In your Admin UI screen (Cloud -> Tree -> configs/myCollection) you should see in right panel below the metadata the content of the ZNode. If it says something else than "Node ... has no utf8 content" then there is something fishy going on...
You can also try fetching the node content from your shell with this command
/opt/solr/server/scripts/cloud-scripts/zkcli.sh -z localhost:9983 -cmd get /configs/foo >foo.txt
It will write the content to "foo.txt", and according to the stack trace, that file will be non-empty and contain non-JSON.
Jan
> 7. nov. 2024 kl. 16:03 skrev Dmitri Maziuk <dm...@gmail.com>:
>
> On 11/7/24 05:31, Jan Høydahl wrote:
>
>> Can you go to Solr Admin UI and inspect what the contents is on your configset ZNode in zookeeper?
>> 
>> Solr expects there to be either no data or a JSON object on the "folder" itself (zookeeper folders can have data). If you for some reason have a non-empty string there which is not JSON this will trip the JSON parsing.
>
> Hmm... what do you mean "inspect ZNode in zookeeper" in "Solr Admin UI", can you be more explicit?
>
> The web GUI has `cloud?view=[tree,nodes,graph,zkstatus]` none of which will show me JSON, in tree view I can see metadata on the /configs/blah folders. ZK has admin UI on port 8080 but I'm not seeing a 4-letter command for fetching a "folder" in TFM. What am I looking for where?
>
> Zookepeer is external, v.3.8.3, w/ 3 ZK nodes but only 2 Solr nodes (num replicas = 2, num shards = 1), if that makes any difference.
>
> Thank you,
> Dima
>
Re: configset api in 8.11?
Posted by Dmitri Maziuk <dm...@gmail.com>.
On 11/7/24 05:31, Jan Høydahl wrote:
> Can you go to Solr Admin UI and inspect what the contents is on your configset ZNode in zookeeper?
> 
>
> Solr expects there to be either no data or a JSON object on the "folder" itself (zookeeper folders can have data). If you for some reason have a non-empty string there which is not JSON this will trip the JSON parsing.
Hmm... what do you mean "inspect ZNode in zookeeper" in "Solr Admin UI",
can you be more explicit?
The web GUI has `cloud?view=[tree,nodes,graph,zkstatus]` none of which
will show me JSON, in tree view I can see metadata on the /configs/blah
folders. ZK has admin UI on port 8080 but I'm not seeing a 4-letter
command for fetching a "folder" in TFM. What am I looking for where?
Zookepeer is external, v.3.8.3, w/ 3 ZK nodes but only 2 Solr nodes (num
replicas = 2, num shards = 1), if that makes any difference.
Thank you,
Dima
Re: configset api in 8.11?
Posted by Jan Høydahl <ja...@cominvent.com>.
Looks like you're tripping on this code line
https://github.com/apache/lucene-solr/blame/branch_8_11/solr/core/src/java/org/apache/solr/handler/admin/ConfigSetsHandler.java#L346
Blaming lucene-solr/solr/core/src/java/org/apache/solr/handler/admin/ConfigSetsHandler.java at branch_8_11 · apache/lucene-solr
github.com
It wants to check that the configset is trusted before allowing you to upload anything, and then it fetches the content of the Zookeeper node for your config set and parses it using a JSON parser.
I spun up a Solr 8.11.2 here, created a config set and tested, no issues.
Can you go to Solr Admin UI and inspect what the contents is on your configset ZNode in zookeeper?

Solr expects there to be either no data or a JSON object on the "folder" itself (zookeeper folders can have data). If you for some reason have a non-empty string there which is not JSON this will trip the JSON parsing.
Jan
> 7. nov. 2024 kl. 01:22 skrev Dmitri Maziuk <dm...@gmail.com>:
>
> On 11/5/24 21:54, Arrieta, Alejandro wrote:
>> Hello Dmitri,
>>>> So my question stands: has anyone actually tried using configset API in
>>>> 8.11, and succeeded?
>> Yes, too many times to count, but just to be sure I tested again now.
>> Solr 8.11.2 with TLS and kerberos
> ...
>
> Interesting... it has to be me, then. I get this on single file upload (full dump attached, may be in BOM encoding):
>
>> 024-11-06 18:23:42.057 INFO (qtp864221358-12202) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/configs params={filePath=solrconfig.xml&name=preview&action=UPLOAD&overwrite=true} status=500 QTime=2
>> 2024-11-06 18:23:42.072 ERROR (qtp864221358-12202) [ ] o.a.s.s.HttpSolrCall org.noggit.JSONParser$ParseException: JSON Parse Error: char=.,position=0 AFTER='.' BEFORE='' => org.noggit.JSONParser$ParseException: JSON Parse Error: char=.,position=0 AFTER='.' BEFORE=''
>> at org.noggit.JSONParser.err(JSONParser.java:452)
>
> "List configsets" works so the endpoint is working, it is an existing config, and -- to Mark's point -- changing Content-type makes no difference.
>
> Our nodes are windows is the only thing I can think of.
>
> zkcli.bat works so this isn't too critical, but... weird.
>
> Thanks,
> Dima
> <service.log>
Re: configset api in 8.11?
Posted by Dmitri Maziuk <dm...@gmail.com>.
On 11/7/24 04:53, David Eric Pugh wrote:
> Would you mind testing this in either 9.7 or main? Make sure that we don't have a bug that needs fixing?
Not anytime soon, sorry: I don't have the infra & can't spin it up
easily with our current provider.
:(
Dima
Re: configset api in 8.11?
Posted by Dmitri Maziuk <dm...@gmail.com>.
On 11/11/24 06:08, Jan Høydahl wrote:
> Cool, there is the reason for null byte. And it is fixed in 8.11.1, good.
According to this:
https://solr.apache.org/docs/8_11_3/changes/Changes.html#v8.11.3.bug_fixes
it's fixed in 8.11.3.
That explains why I am seeing it: we're on 8.11.2 w/ a hand-updated JAR
or two...
Dima
Re: configset api in 8.11?
Posted by Jan Høydahl <ja...@cominvent.com>.
Cool, there is the reason for null byte. And it is fixed in 8.11.1, good. Question is if we want guard rails for people with already messed-up zk content..
Jan Høydahl
> On 11 Nov 2024, at 13:01, Steffen Moldenhauer <s....@intershop.de.invalid> wrote:
>
> Maybe it is something as described in this issue: https://issues.apache.org/jira/browse/SOLR-16110
>
> Regards
> Steffen
>
>> -----Original Message-----
>> From: Jan Høydahl <ja...@cominvent.com>
>> Sent: Monday, November 11, 2024 11:20 AM
>> To: users@solr.apache.org
>> Subject: Re: configset api in 8.11?
>>
>> Hi,
>>
>> I agree that an additional safeguard for what is deemed "empty" could be added
>> here. If you like to attempt a Pull Request, feel free. Or if not, perhaps create a
>> JIRA issue.
>>
>> Jan
>>
>>>> 9. nov. 2024 kl. 20:45 skrev Dmitri Maziuk <dm...@gmail.com>:
>>>
>>> On 11/7/24 04:53, David Eric Pugh wrote:
>>>> Would you mind testing this in either 9.7 or main? Make sure that we don't
>> have a bug that needs fixing?
>>>
>>> @David,
>>>
>>> In case you haven't followed the thread: for some reason there is the 0x00
>> byte where Solr expects "no data or a JSON object" (quoting Jan); it's only in
>> some configsets and we have no idea how it got there.
>>>
>>> Whether it's a bug or not, and whose, it may be worth adding a check before
>> passing the value to JSON parser: the "\0" is "no data"/empty string, at least in
>> traditional C definition. The trace shows it trying to parse the string "\0" (both
>> dots below are 00 in hex):
>>>
>>> org.noggit.JSONParser$ParseException: JSON Parse Error: char=.,position=0
>> AFTER='.' BEFORE='' => org.noggit.JSONParser
>>> $ParseException: JSON Parse Error: char=.,position=0 AFTER='.' BEFORE=''
>>>
>>> That is passed from
>>>
>> org.apache.solr.handler.admin.ConfigSetsHandler.handleConfigUploadRequest(
>> ConfigSetsHandler.java:205)
>>> to
>>> org.apache.solr.common.util.Utils.fromJSON(Utils.java:305)
>>>
>>> It's probably a good idea to check if the string is zero-length somewhere
>> between those two.
>>>
>>> Dima
>>>
>
RE: configset api in 8.11?
Posted by Steffen Moldenhauer <s....@intershop.de.INVALID>.
Maybe it is something as described in this issue: https://issues.apache.org/jira/browse/SOLR-16110
Regards
Steffen
> -----Original Message-----
> From: Jan Høydahl <ja...@cominvent.com>
> Sent: Monday, November 11, 2024 11:20 AM
> To: users@solr.apache.org
> Subject: Re: configset api in 8.11?
>
> Hi,
>
> I agree that an additional safeguard for what is deemed "empty" could be added
> here. If you like to attempt a Pull Request, feel free. Or if not, perhaps create a
> JIRA issue.
>
> Jan
>
> > 9. nov. 2024 kl. 20:45 skrev Dmitri Maziuk <dm...@gmail.com>:
> >
> > On 11/7/24 04:53, David Eric Pugh wrote:
> >> Would you mind testing this in either 9.7 or main? Make sure that we don't
> have a bug that needs fixing?
> >
> > @David,
> >
> > In case you haven't followed the thread: for some reason there is the 0x00
> byte where Solr expects "no data or a JSON object" (quoting Jan); it's only in
> some configsets and we have no idea how it got there.
> >
> > Whether it's a bug or not, and whose, it may be worth adding a check before
> passing the value to JSON parser: the "\0" is "no data"/empty string, at least in
> traditional C definition. The trace shows it trying to parse the string "\0" (both
> dots below are 00 in hex):
> >
> > org.noggit.JSONParser$ParseException: JSON Parse Error: char=.,position=0
> AFTER='.' BEFORE='' => org.noggit.JSONParser
> > $ParseException: JSON Parse Error: char=.,position=0 AFTER='.' BEFORE=''
> >
> > That is passed from
> >
> org.apache.solr.handler.admin.ConfigSetsHandler.handleConfigUploadRequest(
> ConfigSetsHandler.java:205)
> > to
> > org.apache.solr.common.util.Utils.fromJSON(Utils.java:305)
> >
> > It's probably a good idea to check if the string is zero-length somewhere
> between those two.
> >
> > Dima
> >
Re: configset api in 8.11?
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi,
I agree that an additional safeguard for what is deemed "empty" could be added here. If you like to attempt a Pull Request, feel free. Or if not, perhaps create a JIRA issue.
Jan
> 9. nov. 2024 kl. 20:45 skrev Dmitri Maziuk <dm...@gmail.com>:
>
> On 11/7/24 04:53, David Eric Pugh wrote:
>> Would you mind testing this in either 9.7 or main? Make sure that we don't have a bug that needs fixing?
>
> @David,
>
> In case you haven't followed the thread: for some reason there is the 0x00 byte where Solr expects "no data or a JSON object" (quoting Jan); it's only in some configsets and we have no idea how it got there.
>
> Whether it's a bug or not, and whose, it may be worth adding a check before passing the value to JSON parser: the "\0" is "no data"/empty string, at least in traditional C definition. The trace shows it trying to parse the string "\0" (both dots below are 00 in hex):
>
> org.noggit.JSONParser$ParseException: JSON Parse Error: char=.,position=0 AFTER='.' BEFORE='' => org.noggit.JSONParser
> $ParseException: JSON Parse Error: char=.,position=0 AFTER='.' BEFORE=''
>
> That is passed from
> org.apache.solr.handler.admin.ConfigSetsHandler.handleConfigUploadRequest(ConfigSetsHandler.java:205)
> to
> org.apache.solr.common.util.Utils.fromJSON(Utils.java:305)
>
> It's probably a good idea to check if the string is zero-length somewhere between those two.
>
> Dima
>
Re: configset api in 8.11?
Posted by Dmitri Maziuk <dm...@gmail.com>.
On 11/7/24 04:53, David Eric Pugh wrote:
> Would you mind testing this in either 9.7 or main? Make sure that we don't have a bug that needs fixing?
@David,
In case you haven't followed the thread: for some reason there is the
0x00 byte where Solr expects "no data or a JSON object" (quoting Jan);
it's only in some configsets and we have no idea how it got there.
Whether it's a bug or not, and whose, it may be worth adding a check
before passing the value to JSON parser: the "\0" is "no data"/empty
string, at least in traditional C definition. The trace shows it trying
to parse the string "\0" (both dots below are 00 in hex):
org.noggit.JSONParser$ParseException: JSON Parse Error:
char=.,position=0 AFTER='.' BEFORE='' => org.noggit.JSONParser
$ParseException: JSON Parse Error: char=.,position=0 AFTER='.' BEFORE=''
That is passed from
org.apache.solr.handler.admin.ConfigSetsHandler.handleConfigUploadRequest(ConfigSetsHandler.java:205)
to
org.apache.solr.common.util.Utils.fromJSON(Utils.java:305)
It's probably a good idea to check if the string is zero-length
somewhere between those two.
Dima
Re: configset api in 8.11?
Posted by David Eric Pugh <de...@yahoo.com.INVALID>.
Would you mind testing this in either 9.7 or main? Make sure that we don't have a bug that needs fixing?
On Wednesday, November 6, 2024 at 07:23:28 PM EST, Dmitri Maziuk <dm...@gmail.com> wrote:
On 11/5/24 21:54, Arrieta, Alejandro wrote:
> Hello Dmitri,
>
>>> So my question stands: has anyone actually tried using configset API in
>>> 8.11, and succeeded?
>
> Yes, too many times to count, but just to be sure I tested again now.
> Solr 8.11.2 with TLS and kerberos
...
Interesting... it has to be me, then. I get this on single file upload
(full dump attached, may be in BOM encoding):
> 024-11-06 18:23:42.057 INFO (qtp864221358-12202) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/configs params={filePath=solrconfig.xml&name=preview&action=UPLOAD&overwrite=true} status=500 QTime=2
> 2024-11-06 18:23:42.072 ERROR (qtp864221358-12202) [ ] o.a.s.s.HttpSolrCall org.noggit.JSONParser$ParseException: JSON Parse Error: char=.,position=0 AFTER='.' BEFORE='' => org.noggit.JSONParser$ParseException: JSON Parse Error: char=.,position=0 AFTER='.' BEFORE=''
> at org.noggit.JSONParser.err(JSONParser.java:452)
"List configsets" works so the endpoint is working, it is an existing
config, and -- to Mark's point -- changing Content-type makes no difference.
Our nodes are windows is the only thing I can think of.
zkcli.bat works so this isn't too critical, but... weird.
Thanks,
Dima
Re: configset api in 8.11?
Posted by Dmitri Maziuk <dm...@gmail.com>.
On 11/5/24 21:54, Arrieta, Alejandro wrote:
> Hello Dmitri,
>
>>> So my question stands: has anyone actually tried using configset API in
>>> 8.11, and succeeded?
>
> Yes, too many times to count, but just to be sure I tested again now.
> Solr 8.11.2 with TLS and kerberos
...
Interesting... it has to be me, then. I get this on single file upload
(full dump attached, may be in BOM encoding):
> 024-11-06 18:23:42.057 INFO (qtp864221358-12202) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/configs params={filePath=solrconfig.xml&name=preview&action=UPLOAD&overwrite=true} status=500 QTime=2
> 2024-11-06 18:23:42.072 ERROR (qtp864221358-12202) [ ] o.a.s.s.HttpSolrCall org.noggit.JSONParser$ParseException: JSON Parse Error: char=.,position=0 AFTER='.' BEFORE='' => org.noggit.JSONParser$ParseException: JSON Parse Error: char=.,position=0 AFTER='.' BEFORE=''
> at org.noggit.JSONParser.err(JSONParser.java:452)
"List configsets" works so the endpoint is working, it is an existing
config, and -- to Mark's point -- changing Content-type makes no difference.
Our nodes are windows is the only thing I can think of.
zkcli.bat works so this isn't too critical, but... weird.
Thanks,
Dima
Re: configset api in 8.11?
Posted by "Arrieta, Alejandro" <aa...@perrinsoftware.com>.
Hello Dmitri,
>>So my question stands: has anyone actually tried using configset API in
>>8.11, and succeeded?
Yes, too many times to count, but just to be sure I tested again now.
Solr 8.11.2 with TLS and kerberos
https://solr.apache.org/guide/8_11/configsets-api.html#v1uploadconfigset
uploading zip of ranger_audits solr config.
then modified solrconfig.xml on local filesystem and adding my last name to
the comments and uploading with
https://solr.apache.org/guide/8_11/configsets-api.html#v1uploadsinglefile
and my last name appears on solrconfig.xml in zookeeper for that
ranger_audits solr config.
I have never used v2 api.
For the single file upload you need to have the solr config
created/uploaded.
Kind Regards,
Alejandro Arrieta
On Tue, Nov 5, 2024 at 11:03 PM Dmitri Maziuk <dm...@gmail.com>
wrote:
> On 11/5/24 17:42, Jan Høydahl wrote:
> > There are some docs with examples in ref guide
> > Configsets API | Apache Solr Reference Guide 8.11
> > <https://solr.apache.org/guide/8_11/configsets-api.html>
>
> Yes, that was taken pretty much verbatim from The Fine Manual. The "v2"
> option, BTW, also b0rk3d, although with a a different exception: "no
> core retrieved for null".
>
> So my question stands: has anyone actually tried using configset API in
> 8.11, and succeeded?
>
> Dima
>
>
>
Re: configset api in 8.11?
Posted by Dmitri Maziuk <dm...@gmail.com>.
On 11/5/24 17:42, Jan Høydahl wrote:
> There are some docs with examples in ref guide
> Configsets API | Apache Solr Reference Guide 8.11
> <https://solr.apache.org/guide/8_11/configsets-api.html>
Yes, that was taken pretty much verbatim from The Fine Manual. The "v2"
option, BTW, also b0rk3d, although with a a different exception: "no
core retrieved for null".
So my question stands: has anyone actually tried using configset API in
8.11, and succeeded?
Dima
Re: configset api in 8.11?
Posted by Jan Høydahl <ja...@cominvent.com>.
There are some docs with examples in ref guide
[| [Configsets API | Apache Solr Reference Guide
8.11](https://solr.apache.org/guide/8_11/configsets-
api.html)[solr.apache.org](https://solr.apache.org/guide/8_11/configsets-
api.html)|
[](https://solr.apache.org/guide/8_11/configsets-
api.html)
---|---
](https://solr.apache.org/guide/8_11/configsets-api.html)
Jan Høydahl
> On 5 Nov 2024, at 22:45, Dmitri Maziuk <dm...@gmail.com> wrote:
>
>
> Hi all,
>
> anyone knows if
> ```
> curl -X POST -H 'Content-type:application/octet-stream" \
> \--data-binary @solrconfig.xml
> "http://solrnode:8983/solr/admin/configs?action=UPLOAD&name=myindex&filePath=solrconfig.xml&overwrite=true"
> ```
> is supposed to work in 8.11?
>
> I am getting a 500 from `org.noggit.JSONParser$ParseException: JSON Parse
> Error` which is sort of understandable as none of that is JSON... but what
> does JSON parser have to do with anything?
>
> TIA,
> Dima
>