You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Tim Clarke <he...@gmail.com> on 2023/03/18 13:09:32 UTC
docker arguments
Am new to solr and docker, but per
https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-tika.html
I need to pass arguments at some point to solr:
bin/solr start -e schemaless -Dsolr.modules=extraction
do I try to do that at "docker run...." or "docker start...." or once the
container is running (I'm not even sure if that's right verb >.< )
Tim
Re: docker arguments
Posted by Tim Clarke <he...@gmail.com>.
Just got back to this, thanks for your help Shawn. Agree that docker is the
problem here, I don't have time/need for this experiment to faff with
docker so I've gone back to monolithic solr installed in /opt. It's not
going to be in production anyway. I'll move on to other problems. Be useful
if someone finds this thread to post an answer to clear it up.
Tim
On Tue, 21 Mar 2023 at 14:20, Shawn Heisey <ap...@elyograg.org> wrote:
> On 3/20/23 14:42, Tim Clarke wrote:
> > Sorry, I'd stopped it before answering you. I get this after starting:
> >
> > timc@Debian:$ docker exec 9a65798bda0d ps aux | grep solr
>
> Neither the "-e schemaless" nor "-Dsolr.modules=extraction" options were
> honored in the actual solr startup inside the container. I checked
> carefully in the output you shared.
>
> I ran this command after building the 9.3.0-SNAPSHOT docker image:
>
> docker run -p 8983:8983 -e SOLR_MODULES=extraction -e SOLR_HEAP=4g
> apache/solr:9.3.0-SNAPSHOT
>
> And this was the relevant line from the "docker exec" command:
>
> solr 14 141 4.9 12732896 4528244 ? Sl 22:26 0:23
> /opt/java/openjdk/bin/java -server -Xms4g -Xmx4g -XX:+UseG1GC
> -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled
> -XX:MaxGCPauseMillis=250 -XX:+UseLargePages -XX:+AlwaysPreTouch
> -XX:+ExplicitGCInvokesConcurrent
> -Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
>
> -Dsolr.jetty.inetaccess.includes= -Dsolr.jetty.inetaccess.excludes=
> -Dsolr.log.dir=/var/solr/logs -Djetty.port=8983 -DSTOP.PORT=7983
> -DSTOP.KEY=solrrocks -Duser.timezone=UTC -XX:-OmitStackTraceInFastThrow
> -XX:+CrashOnOutOfMemoryError
> -XX:ErrorFile=/var/solr/logs/jvm_crash_%p.log
> -Djetty.home=/opt/solr-9.3.0-SNAPSHOT/server
> -Dsolr.solr.home=/var/solr/data
> -Dsolr.install.dir=/opt/solr-9.3.0-SNAPSHOT
> -Dsolr.install.symDir=/opt/solr
> -Dsolr.default.confdir=/opt/solr-9.3.0-SNAPSHOT/server/solr/configsets/_default/conf
>
> -Dlog4j.configurationFile=/var/solr/log4j2.xml -Dsolr.modules=extraction
> -Dsolr.jetty.host=0.0.0.0 -Xss256k
> -XX:CompileCommand=exclude,com.github.benmanes.caffeine.cache.BoundedLocalCache::put
>
> -Djava.security.manager
> -Djava.security.policy=/opt/solr-9.3.0-SNAPSHOT/server/etc/security.policy
> -Djava.security.properties=/opt/solr-9.3.0-SNAPSHOT/server/etc/security.properties
>
> -Dsolr.internal.network.permission=* -DdisableAdminUI=false -jar
> start.jar --module=http --module=requestlog --module=gzip
>
> You can see that the two environment variables I set did affect the
> startup commandline.
>
> I think the problem with the docker run command that you used is that
> the options you provided were handled by docker, and not sent as-is to
> the Solr start script inside the container.
>
> Just in the last few days, I have encountered evidence that some of what
> the ref guide shows for running the docker image is not actually
> supported by the docker image. Maybe there are more things wrong.
>
> I don't know that there is actually a way to have the docker image
> easily run an example. You wouldn't want to run an example in
> production anyway.
>
> Something you can do is just use the config from the schemaless example
> when you create a new collection.
>
> Thanks,
> Shawn
>
Re: docker arguments
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/20/23 14:42, Tim Clarke wrote:
> Sorry, I'd stopped it before answering you. I get this after starting:
>
> timc@Debian:$ docker exec 9a65798bda0d ps aux | grep solr
Neither the "-e schemaless" nor "-Dsolr.modules=extraction" options were
honored in the actual solr startup inside the container. I checked
carefully in the output you shared.
I ran this command after building the 9.3.0-SNAPSHOT docker image:
docker run -p 8983:8983 -e SOLR_MODULES=extraction -e SOLR_HEAP=4g
apache/solr:9.3.0-SNAPSHOT
And this was the relevant line from the "docker exec" command:
solr 14 141 4.9 12732896 4528244 ? Sl 22:26 0:23
/opt/java/openjdk/bin/java -server -Xms4g -Xmx4g -XX:+UseG1GC
-XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled
-XX:MaxGCPauseMillis=250 -XX:+UseLargePages -XX:+AlwaysPreTouch
-XX:+ExplicitGCInvokesConcurrent
-Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
-Dsolr.jetty.inetaccess.includes= -Dsolr.jetty.inetaccess.excludes=
-Dsolr.log.dir=/var/solr/logs -Djetty.port=8983 -DSTOP.PORT=7983
-DSTOP.KEY=solrrocks -Duser.timezone=UTC -XX:-OmitStackTraceInFastThrow
-XX:+CrashOnOutOfMemoryError
-XX:ErrorFile=/var/solr/logs/jvm_crash_%p.log
-Djetty.home=/opt/solr-9.3.0-SNAPSHOT/server
-Dsolr.solr.home=/var/solr/data
-Dsolr.install.dir=/opt/solr-9.3.0-SNAPSHOT
-Dsolr.install.symDir=/opt/solr
-Dsolr.default.confdir=/opt/solr-9.3.0-SNAPSHOT/server/solr/configsets/_default/conf
-Dlog4j.configurationFile=/var/solr/log4j2.xml -Dsolr.modules=extraction
-Dsolr.jetty.host=0.0.0.0 -Xss256k
-XX:CompileCommand=exclude,com.github.benmanes.caffeine.cache.BoundedLocalCache::put
-Djava.security.manager
-Djava.security.policy=/opt/solr-9.3.0-SNAPSHOT/server/etc/security.policy
-Djava.security.properties=/opt/solr-9.3.0-SNAPSHOT/server/etc/security.properties
-Dsolr.internal.network.permission=* -DdisableAdminUI=false -jar
start.jar --module=http --module=requestlog --module=gzip
You can see that the two environment variables I set did affect the
startup commandline.
I think the problem with the docker run command that you used is that
the options you provided were handled by docker, and not sent as-is to
the Solr start script inside the container.
Just in the last few days, I have encountered evidence that some of what
the ref guide shows for running the docker image is not actually
supported by the docker image. Maybe there are more things wrong.
I don't know that there is actually a way to have the docker image
easily run an example. You wouldn't want to run an example in
production anyway.
Something you can do is just use the config from the schemaless example
when you create a new collection.
Thanks,
Shawn
Re: docker arguments
Posted by Tim Clarke <he...@gmail.com>.
Sorry, I'd stopped it before answering you. I get this after starting:
timc@Debian:$ docker exec 9a65798bda0d ps aux | grep solr
solr 1 0.0 0.0 2504 524 ? Ss 20:34 0:00 tini --
solr -f
solr 12 11.7 5.0 5424836 715596 ? Sl 20:34 0:14
/opt/java/openjdk/bin/java -server -Xms512m -Xmx512m -XX:+UseG1GC
-XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled
-XX:MaxGCPauseMillis=250 -XX:+UseLargePages -XX:+AlwaysPreTouch
-XX:+ExplicitGCInvokesConcurrent
-Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
-Dsolr.jetty.inetaccess.includes= -Dsolr.jetty.inetaccess.excludes=
-Dsolr.log.dir=/var/solr/logs -Djetty.port=8983 -DSTOP.PORT=7983
-DSTOP.KEY=solrrocks -Duser.timezone=UTC -XX:-OmitStackTraceInFastThrow
-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /var/solr/logs
-Djetty.home=/opt/solr/server -Dsolr.solr.home=/var/solr/data
-Dsolr.data.home= -Dsolr.install.dir=/opt/solr
-Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf
-Dlog4j.configurationFile=/var/solr/log4j2.xml -Dsolr.jetty.host=0.0.0.0
-Xss256k
-XX:CompileCommand=exclude,com.github.benmanes.caffeine.cache.BoundedLocalCache::put
-Djava.security.manager
-Djava.security.policy=/opt/solr/server/etc/security.policy
-Djava.security.properties=/opt/solr/server/etc/security.properties
-Dsolr.internal.network.permission=* -DdisableAdminUI=false -jar start.jar
--module=http --module=requestlog --module=gzip
solr 119 0.0 0.0 8896 3220 ? Rs 20:36 0:00 ps aux
timc@Debian:$
Tim
On Sun, 19 Mar 2023 at 14:56, Shawn Heisey <ap...@elyograg.org> wrote:
> On 3/18/23 12:21, Tim Clarke wrote:
> > CONTAINER ID IMAGE COMMAND CREATED
> STATUS
> > PORTS NAMES
> > 3eebcc1a2b25 solr "docker-entrypoint.s…" 5 hours ago
> Exited
> > (137) 23 minutes ago my_solr2
> > 9a65798bda0d solr "docker-entrypoint.s…" 9 days ago
> Exited
> > (143) 5 hours ago my_solr
> > 25dd9eee328b hello-world "/hello" 9 days ago
> Exited
> > (0) 9 days ago cool_davinci
> > 8b5591632935 hello-world "/hello" 9 days ago
> Exited
> > (0) 9 days ago vigilant_ride
> >
> > (The top solr is the container I'm running with the additional " -e
> > schemaless -Dsolr.modules=extraction" arguments per the solr set-up page
> > previously cited.
>
> It says none of those containers is running. When you have one that IS
> running, use its ID value in a command like this:
>
> docker exec 3eebcc1a2b25 ps auxw | grep solr
>
> And send us the output. Paste it into the message. If you make it an
> attachment, the mailing list is likely to eat it so we never see it.
>
> Thanks,
> Shawn
>
Re: docker arguments
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/18/23 12:21, Tim Clarke wrote:
> CONTAINER ID IMAGE COMMAND CREATED STATUS
> PORTS NAMES
> 3eebcc1a2b25 solr "docker-entrypoint.s…" 5 hours ago Exited
> (137) 23 minutes ago my_solr2
> 9a65798bda0d solr "docker-entrypoint.s…" 9 days ago Exited
> (143) 5 hours ago my_solr
> 25dd9eee328b hello-world "/hello" 9 days ago Exited
> (0) 9 days ago cool_davinci
> 8b5591632935 hello-world "/hello" 9 days ago Exited
> (0) 9 days ago vigilant_ride
>
> (The top solr is the container I'm running with the additional " -e
> schemaless -Dsolr.modules=extraction" arguments per the solr set-up page
> previously cited.
It says none of those containers is running. When you have one that IS
running, use its ID value in a command like this:
docker exec 3eebcc1a2b25 ps auxw | grep solr
And send us the output. Paste it into the message. If you make it an
attachment, the mailing list is likely to eat it so we never see it.
Thanks,
Shawn
Re: docker arguments
Posted by Tim Clarke <he...@gmail.com>.
"docker run -d -v "/home/timc/dev//solrdata:/var/solr" -p 8983:8983 --name
my_solr solr solr-precreate gettingstarted -e schemaless
-Dsolr.modules=extraction"
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS
PORTS NAMES
3eebcc1a2b25 solr "docker-entrypoint.s…" 5 hours ago Exited
(137) 23 minutes ago my_solr2
9a65798bda0d solr "docker-entrypoint.s…" 9 days ago Exited
(143) 5 hours ago my_solr
25dd9eee328b hello-world "/hello" 9 days ago Exited
(0) 9 days ago cool_davinci
8b5591632935 hello-world "/hello" 9 days ago Exited
(0) 9 days ago vigilant_ride
(The top solr is the container I'm running with the additional " -e
schemaless -Dsolr.modules=extraction" arguments per the solr set-up page
previously cited.
Tim
On Sat, 18 Mar 2023 at 18:04, Shawn Heisey <ap...@elyograg.org> wrote:
> On 3/18/23 07:34, Tim Clarke wrote:
> > Hmmm, I may have made progress by passing the arguments at the "docker
> run"
> > stage:
> >
> > But when the curl command is issued per to add the request handler, I
> > receive this reply:
>
> <snip>
>
> > "errorMessages":[" Error loading class
> > 'solr.extraction.ExtractingRequestHandler'"]}],
>
> It can't find the class. So I think the module must not be loaded.
>
> Exactly what 'docker run' command did you use? What is the output of
> "docker ps -a"?
>
> Thanks,
> Shawn
>
Re: docker arguments
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/18/23 07:34, Tim Clarke wrote:
> Hmmm, I may have made progress by passing the arguments at the "docker run"
> stage:
>
> But when the curl command is issued per to add the request handler, I
> receive this reply:
<snip>
> "errorMessages":[" Error loading class
> 'solr.extraction.ExtractingRequestHandler'"]}],
It can't find the class. So I think the module must not be loaded.
Exactly what 'docker run' command did you use? What is the output of
"docker ps -a"?
Thanks,
Shawn
Re: docker arguments
Posted by dmitri maziuk <dm...@gmail.com>.
On 2023-03-20 5:29 PM, Tim Clarke wrote:
> And that's a good lead - thanks - it contains the elusive solrconfig.xml
> file which I'd not been finding since I unexpectedly had no read rights to
> the folder.
Yeah, right: don't never ever run as root. Except when you actually need to.
Dima
Re: docker arguments
Posted by Tim Clarke <he...@gmail.com>.
The example instruction I'm following uses this line:
docker exec -it my_solr post -c gettingstarted
example/exampledocs/manufacturers.xml
which sets up the folder
~/dev/solrdata/data/gettingstarted
And that's a good lead - thanks - it contains the elusive solrconfig.xml
file which I'd not been finding since I unexpectedly had no read rights to
the folder.
On Mon, 20 Mar 2023 at 20:52, dmitri maziuk <dm...@gmail.com> wrote:
> On 2023-03-20 3:33 PM, Tim Clarke wrote:
> > Thanks Dima, that FM reference is the 8.x version of the 9.x one I've
> been
> > following :D
> >
> https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-tika.html
> >
> > But there's no solrconfig.xml in that data folder - it must be inside the
> > docker image/container I think.
>
> We're still on 8x so that what I have in my bookmarks.
>
> If the container is actually running, creating a core should set it all
> up incl. solrconfig.xml -- presumably somewhere under your
> /home/timc/dev/solrdata
>
> So what happens when you create the core for your documents?
>
> Dima
>
>
Re: docker arguments
Posted by dmitri maziuk <dm...@gmail.com>.
On 2023-03-20 3:33 PM, Tim Clarke wrote:
> Thanks Dima, that FM reference is the 8.x version of the 9.x one I've been
> following :D
> https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-tika.html
>
> But there's no solrconfig.xml in that data folder - it must be inside the
> docker image/container I think.
We're still on 8x so that what I have in my bookmarks.
If the container is actually running, creating a core should set it all
up incl. solrconfig.xml -- presumably somewhere under your
/home/timc/dev/solrdata
So what happens when you create the core for your documents?
Dima
Re: docker arguments
Posted by Tim Clarke <he...@gmail.com>.
Thanks Dima, that FM reference is the 8.x version of the 9.x one I've been
following :D
https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-tika.html
But there's no solrconfig.xml in that data folder - it must be inside the
docker image/container I think.
Tim
On Sat, 18 Mar 2023 at 18:37, dmitri maziuk <dm...@gmail.com> wrote:
> On 2023-03-18 12:17 PM, Tim Clarke wrote:
> > Presumably that would be the CLASSPATH inside the docker image and/or
> the
> > libraries available within the image? I'd have to find out how to check
> > that. Not really a question for here, but does a docker image have a
> source
> > to check?
>
> I can't tell you about the container you're using, but in general
> Dockerfile and `docker exec ... /bin/sh` are where to look.
>
> Where's solrconfig.xml coming from, is it in your
> /home/timc/dev/solrdata ? -- maybe start there., check it against TFM
> link I sent earlier ... this:
>
> https://solr.apache.org/guide/8_11/uploading-data-with-solr-cell-using-apache-tika.html#configuring-the-extractingrequesthandler-in-solrconfig-xml
>
> Dima
>
>
Re: docker arguments
Posted by dmitri maziuk <dm...@gmail.com>.
On 2023-03-18 12:17 PM, Tim Clarke wrote:
> Presumably that would be the CLASSPATH inside the docker image and/or the
> libraries available within the image? I'd have to find out how to check
> that. Not really a question for here, but does a docker image have a source
> to check?
I can't tell you about the container you're using, but in general
Dockerfile and `docker exec ... /bin/sh` are where to look.
Where's solrconfig.xml coming from, is it in your
/home/timc/dev/solrdata ? -- maybe start there., check it against TFM
link I sent earlier ... this:
https://solr.apache.org/guide/8_11/uploading-data-with-solr-cell-using-apache-tika.html#configuring-the-extractingrequesthandler-in-solrconfig-xml
Dima
Re: docker arguments
Posted by Tim Clarke <he...@gmail.com>.
Presumably that would be the CLASSPATH inside the docker image and/or the
libraries available within the image? I'd have to find out how to check
that. Not really a question for here, but does a docker image have a source
to check?
Tim
On Sat, 18 Mar 2023 at 16:06, dmitri maziuk <dm...@gmail.com> wrote:
> On 2023-03-18 8:34 AM, Tim Clarke wrote:
>
> > errorMessages=[ Error loading class
> > 'solr.extraction.ExtractingRequestHandler']}], \n"],
>
> It's not in CLASSPATH presumably.
>
>
> https://solr.apache.org/guide/8_11/uploading-data-with-solr-cell-using-apache-tika.html#configuring-the-extractingrequesthandler-in-solrconfig-xml
>
> Dima
>
>
Re: docker arguments
Posted by dmitri maziuk <dm...@gmail.com>.
On 2023-03-18 8:34 AM, Tim Clarke wrote:
> errorMessages=[ Error loading class
> 'solr.extraction.ExtractingRequestHandler']}], \n"],
It's not in CLASSPATH presumably.
https://solr.apache.org/guide/8_11/uploading-data-with-solr-cell-using-apache-tika.html#configuring-the-extractingrequesthandler-in-solrconfig-xml
Dima
Re: docker arguments
Posted by Tim Clarke <he...@gmail.com>.
Hmmm, I may have made progress by passing the arguments at the "docker run"
stage:
But when the curl command is issued per to add the request handler, I
receive this reply:
timc@Debian$ curl -X POST -H 'Content-type:application/json' -d '{
"add-requesthandler": {
"name": "/update/extract",
"class": "solr.extraction.ExtractingRequestHandler",
"defaults":{ "lowernames": "true", "captureAttr":"true"}
}
}' 'http://localhost:8983/solr/gettingstarted/config'
{
"responseHeader":{
"status":400,
"QTime":24},
"errorMessages":["error processing commands, errors:
[{add-requesthandler={name=/update/extract,
class=solr.extraction.ExtractingRequestHandler, defaults={lowernames=true,
captureAttr=true}}, errorMessages=[ Error loading class
'solr.extraction.ExtractingRequestHandler']}], \n"],
"WARNING":"This response format is experimental. It is likely to change
in the future.",
"error":{
"metadata":[
"error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject",
"root-error-class","org.apache.solr.api.ApiBag$ExceptionWithErrObject"],
"details":[{
"add-requesthandler":{
"name":"/update/extract",
"class":"solr.extraction.ExtractingRequestHandler",
"defaults":{
"lowernames":"true",
"captureAttr":"true"}},
"errorMessages":[" Error loading class
'solr.extraction.ExtractingRequestHandler'"]}],
"msg":"error processing commands, errors:
[{add-requesthandler={name=/update/extract,
class=solr.extraction.ExtractingRequestHandler, defaults={lowernames=true,
captureAttr=true}}, errorMessages=[ Error loading class
'solr.extraction.ExtractingRequestHandler']}], ",
"code":400}}
Any ideas, anyone?
Tim
On Sat, 18 Mar 2023 at 13:09, Tim Clarke <he...@gmail.com> wrote:
> Am new to solr and docker, but per
> https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-tika.html
> I need to pass arguments at some point to solr:
>
> bin/solr start -e schemaless -Dsolr.modules=extraction
>
> do I try to do that at "docker run...." or "docker start...." or once the
> container is running (I'm not even sure if that's right verb >.< )
>
> Tim
>