You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Ryan Work <ry...@gmail.com> on 2022/02/09 21:21:44 UTC
Help! Can't Get Solr 8.11.1 to Work
I am trying to set up Solr 8.11.1 on a new server that is running RHEL 8
along with Java 11 openjdk. I followed all of the steps listed in the Solr
guide (https://solr.apache.org/guide/8_11/taking-solr-to-production.html)
I pretty much was able to follow the guide with no issue up until the part
where it checks the progress (where I'm supposed to start solr). That's
when I try to start Solr and see the error
"java.lang.IllegalStateException: /opt/solr-8.11.1/server/NONE is not a
valid keystore"..."Caused by: javax.servlet.UnavailableException: Error
processing the request. CoreContainer is either not initialized or shutting
down."
I ran the install script that comes packaged with Solr and which the guide
says to run. I kept all the default values. So I'm not sure why I'm getting
this issue. The odd thing is that I was able to install Solr 7.7 with
similar steps provided on Solr's site and got it to work with no issues. So
only Solr 8 is not working for me. I've tried a few of the Solr 8.X
versions, but all gave me the same error and the logs aren't providing any
other relevant message besides the ones I mentioned above. Any ideas on why
I'm getting it and how to resolve it? Thank you in advance!
Re: Help! Can't Get Solr 8.11.1 to Work
Posted by Ryan Work <ry...@gmail.com>.
Hi,
I've got some new updates on this issue. We found out that the host we are
using has SELinux enabled. It is also a FIPS enabled environment. So we had
one of our admins apply the necessary SELinux settings to allow Solr which
got rid of the SELinux errors we saw. Now we're trying to figure out how to
get Solr to work in a FIPS enabled environment, but have no clue on how to
do that. We've tried looking online, but haven't come across anything
helpful. Does anyone here have any experience or advice on how to get Solr
to work in a FIPS enabled environment?
Thank you in advance for your help!
On Mon, Feb 14, 2022 at 3:55 PM Ryan Work <ry...@gmail.com> wrote:
> Hi,
>
> I've checked my solr user environment variables, but didn't find any SSL
> settings for Solr. I've also checked my init.d script and verified that the
> one I'm expecting it to use is in fact the one Solr is using by changing a
> variable within the script and seeing if Solr picked it up which it did. So
> I know that the solr.in.sh file I provided earlier is what is being used.
>
> I also, ran the command you provided, "ps -efwww | grep solr", and got the
> following output that Solr is running with:
> -server
> -Xms512m
> -Xmx512m
> -XX:+UseG1GC
> -XX:+PerfDisableSharedMem
> -XX:+ParallelRefProcEnabled
> -XX:MaxGCPauseMillis=250
> -XX:+UseLargePages
> -XX:+AlwaysPreTouch
> -XX:+ExplicitGCInvokesConcurrent
>
> -Xlog:gc*:file=/search/solr_data/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
> -Dsolr.jetty.inetaccess.includes=
> -Dsolr.jetty.inetaccess.excludes=
> -Dsolr.log.dir=/search/solr_data/logs
> -Djetty.port=8983
> -DSTOP.PORT=7983
> -DSTOP.KEY=solrrocks
> -Duser.timezone=UTC
> -XX:-OmitStackTraceInFastThrow
> -XX:OnOutOfMemoryError=/search/solr/bin/oom_solr.sh 8983
> /search/solr_data/logs
> -Djetty.home=/search/solr/server
> -Dsolr.solr.home=/search/solr_data/data
> -Dsolr.data.home=
> -Dsolr.install.dir=/search/solr
> -Dsolr.default.confdir=/search/solr/server/solr/configsets/_default/conf
> -Dlog4j.configurationFile=/search/solr_data/log4j2.xml
> -Xss256k
> -Dsolr.log.muteconsole
> -jar start.jar
> --module=http
> --module=gzip
>
>
> What still intrigues me is that if I uninstall Solr 8.11.1 and then
> install Solr 7.7 and follow the same install steps, Solr 7.7 works with no
> issue. Any other ideas?
>
> On Thu, Feb 10, 2022 at 5:58 PM Jan Høydahl <ja...@cominvent.com> wrote:
>
>> Hi
>>
>> Somehow you have initialized the environment with SSL settings. I see
>> your question on StackOverflow too
>> <https://stackoverflow.com/questions/70962385/solr-8-not-working-when-trying-to-start-solr-java-lang-illegalstateexception>,
>> where you explicitly use SSL and install to another location.
>> So please check your "solr" user whether perhaps some env.variables are
>> left (run "export" and look for SOLR_SSL_*).
>> Also check your init.d or systemd init script whether it sets any
>> variables or perhaps choose a different solr.in.sh file than you think.
>>
>> Finally, you can inspect exactly what command line the solr instance is
>> started with, by issuing a command like this on the host:
>>
>> ps -efwww | grep solr
>>
>> You should look for things like -Djavax.net.ssl.keyStore
>>
>> It looks like you have initialized environemtn SOLR_SSL_KEY_STORE=NONE
>>
>> Jan
>>
>> 10. feb. 2022 kl. 22:01 skrev Ryan Work <ry...@gmail.com>:
>>
>> Hi,
>>
>> I actually left my /etc/default/solr.in.sh file alone along with the
>> default settings it came with. The only changes I've made to the default
>> settings is replacing the /opt and /var directories when installing Solr
>> using the install script (step 3d below). Other than that I've left
>> everything else as default. Also, I have not done any steps to enable Solr
>> for SSL.
>>
>> I've attached my solr.in.sh file, the solr.log file, and the solr status
>> output for reference. I've also included the steps I took below to see if
>> anyone can reproduce my issue.
>>
>> OS: Red Hat Enterprise Linux release 8.5 (Ootpa)
>>
>> 1. Install the Java openjdk:
>> yum install java-11-openjdk
>>
>> 2. Install lsof:
>> yum install lsof
>>
>>
>> 3. Install and Configure Solr:
>> 3a. Download Solr to the working directory (/search)
>> cd /search
>> wget https://downloads.apache.org/lucene/solr/8.11.1/solr-8.11.1.tgz
>>
>> 3b. Make the Solr Data directory:
>> mkdir /search/solr_data
>>
>> 3c.Extract the Solr install script from the .tgz file:
>> tar xzf solr-8.11.1.tgz solr-8.11.1/bin/install_solr_service.sh
>> --strip-components=2
>>
>> 3d. Run the Solr install script as root (replace the /opt/solr and
>> /var/solr directories with /search and /search/solr_data respectively):
>> ./install_solr_service.sh solr-8.11.1.tgz -i /search -d /search/solr_data
>> -u solr -s solr -p 8983
>>
>> Once Solr starts it looks like it starts and displays the "Happy
>> searching" message, but when I run a service solr status, I get the
>> attached output.
>>
>>
>>
>> On Thu, Feb 10, 2022 at 3:27 AM Jan Høydahl <ja...@cominvent.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Please see in your /etc/default/solr.in.sh file whether you have any
>>> SOLR_SSL_* variables defined and if those are correctly set up.
>>>
>>> If you cannot find anything, it would help a lot if you would provide
>>> the entire log from /var/solr/data/logs/solr.log as well as the entire
>>> /etc/default/solr.in.sh file, for debugging.
>>>
>>> Jan
>>>
>>> 9. feb. 2022 kl. 22:21 skrev Ryan Work <ry...@gmail.com>:
>>>
>>> I am trying to set up Solr 8.11.1 on a new server that is running RHEL 8
>>> along with Java 11 openjdk. I followed all of the steps listed in the
>>> Solr
>>> guide (https://solr.apache.org/guide/8_11/taking-solr-to-production.html
>>> )
>>>
>>> I pretty much was able to follow the guide with no issue up until the
>>> part
>>> where it checks the progress (where I'm supposed to start solr). That's
>>> when I try to start Solr and see the error
>>> "java.lang.IllegalStateException: /opt/solr-8.11.1/server/NONE is not a
>>> valid keystore"..."Caused by: javax.servlet.UnavailableException: Error
>>> processing the request. CoreContainer is either not initialized or
>>> shutting
>>> down."
>>>
>>> I ran the install script that comes packaged with Solr and which the
>>> guide
>>> says to run. I kept all the default values. So I'm not sure why I'm
>>> getting
>>> this issue. The odd thing is that I was able to install Solr 7.7 with
>>> similar steps provided on Solr's site and got it to work with no issues.
>>> So
>>> only Solr 8 is not working for me. I've tried a few of the Solr 8.X
>>> versions, but all gave me the same error and the logs aren't providing
>>> any
>>> other relevant message besides the ones I mentioned above. Any ideas on
>>> why
>>> I'm getting it and how to resolve it? Thank you in advance!
>>>
>>>
>>> <solr_status_output.txt><solr.log><solr.in.sh>
>>
>>
>>
Re: Help! Can't Get Solr 8.11.1 to Work
Posted by Ryan Work <ry...@gmail.com>.
Hi,
I've checked my solr user environment variables, but didn't find any SSL
settings for Solr. I've also checked my init.d script and verified that the
one I'm expecting it to use is in fact the one Solr is using by changing a
variable within the script and seeing if Solr picked it up which it did. So
I know that the solr.in.sh file I provided earlier is what is being used.
I also, ran the command you provided, "ps -efwww | grep solr", and got the
following output that Solr is running with:
-server
-Xms512m
-Xmx512m
-XX:+UseG1GC
-XX:+PerfDisableSharedMem
-XX:+ParallelRefProcEnabled
-XX:MaxGCPauseMillis=250
-XX:+UseLargePages
-XX:+AlwaysPreTouch
-XX:+ExplicitGCInvokesConcurrent
-Xlog:gc*:file=/search/solr_data/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
-Dsolr.jetty.inetaccess.includes=
-Dsolr.jetty.inetaccess.excludes=
-Dsolr.log.dir=/search/solr_data/logs
-Djetty.port=8983
-DSTOP.PORT=7983
-DSTOP.KEY=solrrocks
-Duser.timezone=UTC
-XX:-OmitStackTraceInFastThrow
-XX:OnOutOfMemoryError=/search/solr/bin/oom_solr.sh 8983
/search/solr_data/logs
-Djetty.home=/search/solr/server
-Dsolr.solr.home=/search/solr_data/data
-Dsolr.data.home=
-Dsolr.install.dir=/search/solr
-Dsolr.default.confdir=/search/solr/server/solr/configsets/_default/conf
-Dlog4j.configurationFile=/search/solr_data/log4j2.xml
-Xss256k
-Dsolr.log.muteconsole
-jar start.jar
--module=http
--module=gzip
What still intrigues me is that if I uninstall Solr 8.11.1 and then install
Solr 7.7 and follow the same install steps, Solr 7.7 works with no issue.
Any other ideas?
On Thu, Feb 10, 2022 at 5:58 PM Jan Høydahl <ja...@cominvent.com> wrote:
> Hi
>
> Somehow you have initialized the environment with SSL settings. I see your
> question on StackOverflow too
> <https://stackoverflow.com/questions/70962385/solr-8-not-working-when-trying-to-start-solr-java-lang-illegalstateexception>,
> where you explicitly use SSL and install to another location.
> So please check your "solr" user whether perhaps some env.variables are
> left (run "export" and look for SOLR_SSL_*).
> Also check your init.d or systemd init script whether it sets any
> variables or perhaps choose a different solr.in.sh file than you think.
>
> Finally, you can inspect exactly what command line the solr instance is
> started with, by issuing a command like this on the host:
>
> ps -efwww | grep solr
>
> You should look for things like -Djavax.net.ssl.keyStore
>
> It looks like you have initialized environemtn SOLR_SSL_KEY_STORE=NONE
>
> Jan
>
> 10. feb. 2022 kl. 22:01 skrev Ryan Work <ry...@gmail.com>:
>
> Hi,
>
> I actually left my /etc/default/solr.in.sh file alone along with the
> default settings it came with. The only changes I've made to the default
> settings is replacing the /opt and /var directories when installing Solr
> using the install script (step 3d below). Other than that I've left
> everything else as default. Also, I have not done any steps to enable Solr
> for SSL.
>
> I've attached my solr.in.sh file, the solr.log file, and the solr status
> output for reference. I've also included the steps I took below to see if
> anyone can reproduce my issue.
>
> OS: Red Hat Enterprise Linux release 8.5 (Ootpa)
>
> 1. Install the Java openjdk:
> yum install java-11-openjdk
>
> 2. Install lsof:
> yum install lsof
>
>
> 3. Install and Configure Solr:
> 3a. Download Solr to the working directory (/search)
> cd /search
> wget https://downloads.apache.org/lucene/solr/8.11.1/solr-8.11.1.tgz
>
> 3b. Make the Solr Data directory:
> mkdir /search/solr_data
>
> 3c.Extract the Solr install script from the .tgz file:
> tar xzf solr-8.11.1.tgz solr-8.11.1/bin/install_solr_service.sh
> --strip-components=2
>
> 3d. Run the Solr install script as root (replace the /opt/solr and
> /var/solr directories with /search and /search/solr_data respectively):
> ./install_solr_service.sh solr-8.11.1.tgz -i /search -d /search/solr_data
> -u solr -s solr -p 8983
>
> Once Solr starts it looks like it starts and displays the "Happy
> searching" message, but when I run a service solr status, I get the
> attached output.
>
>
>
> On Thu, Feb 10, 2022 at 3:27 AM Jan Høydahl <ja...@cominvent.com> wrote:
>
>> Hi,
>>
>> Please see in your /etc/default/solr.in.sh file whether you have any
>> SOLR_SSL_* variables defined and if those are correctly set up.
>>
>> If you cannot find anything, it would help a lot if you would provide the
>> entire log from /var/solr/data/logs/solr.log as well as the entire
>> /etc/default/solr.in.sh file, for debugging.
>>
>> Jan
>>
>> 9. feb. 2022 kl. 22:21 skrev Ryan Work <ry...@gmail.com>:
>>
>> I am trying to set up Solr 8.11.1 on a new server that is running RHEL 8
>> along with Java 11 openjdk. I followed all of the steps listed in the Solr
>> guide (https://solr.apache.org/guide/8_11/taking-solr-to-production.html)
>>
>> I pretty much was able to follow the guide with no issue up until the part
>> where it checks the progress (where I'm supposed to start solr). That's
>> when I try to start Solr and see the error
>> "java.lang.IllegalStateException: /opt/solr-8.11.1/server/NONE is not a
>> valid keystore"..."Caused by: javax.servlet.UnavailableException: Error
>> processing the request. CoreContainer is either not initialized or
>> shutting
>> down."
>>
>> I ran the install script that comes packaged with Solr and which the guide
>> says to run. I kept all the default values. So I'm not sure why I'm
>> getting
>> this issue. The odd thing is that I was able to install Solr 7.7 with
>> similar steps provided on Solr's site and got it to work with no issues.
>> So
>> only Solr 8 is not working for me. I've tried a few of the Solr 8.X
>> versions, but all gave me the same error and the logs aren't providing any
>> other relevant message besides the ones I mentioned above. Any ideas on
>> why
>> I'm getting it and how to resolve it? Thank you in advance!
>>
>>
>> <solr_status_output.txt><solr.log><solr.in.sh>
>
>
>
Re: Help! Can't Get Solr 8.11.1 to Work
Posted by Ryan Work <ry...@gmail.com>.
Hi,
I actually left my /etc/default/solr.in.sh file alone along with the
default settings it came with. The only changes I've made to the default
settings is replacing the /opt and /var directories when installing Solr
using the install script (step 3d below). Other than that I've left
everything else as default. Also, I have not done any steps to enable Solr
for SSL.
I've attached my solr.in.sh file, the solr.log file, and the solr status
output for reference. I've also included the steps I took below to see if
anyone can reproduce my issue.
OS: Red Hat Enterprise Linux release 8.5 (Ootpa)
1. Install the Java openjdk:
yum install java-11-openjdk
2. Install lsof:
yum install lsof
3. Install and Configure Solr:
3a. Download Solr to the working directory (/search)
cd /search
wget https://downloads.apache.org/lucene/solr/8.11.1/solr-8.11.1.tgz
3b. Make the Solr Data directory:
mkdir /search/solr_data
3c.Extract the Solr install script from the .tgz file:
tar xzf solr-8.11.1.tgz solr-8.11.1/bin/install_solr_service.sh
--strip-components=2
3d. Run the Solr install script as root (replace the /opt/solr and
/var/solr directories with /search and /search/solr_data respectively):
./install_solr_service.sh solr-8.11.1.tgz -i /search -d /search/solr_data
-u solr -s solr -p 8983
Once Solr starts it looks like it starts and displays the "Happy searching"
message, but when I run a service solr status, I get the attached output.
On Thu, Feb 10, 2022 at 3:27 AM Jan Høydahl <ja...@cominvent.com> wrote:
> Hi,
>
> Please see in your /etc/default/solr.in.sh file whether you have any
> SOLR_SSL_* variables defined and if those are correctly set up.
>
> If you cannot find anything, it would help a lot if you would provide the
> entire log from /var/solr/data/logs/solr.log as well as the entire
> /etc/default/solr.in.sh file, for debugging.
>
> Jan
>
> 9. feb. 2022 kl. 22:21 skrev Ryan Work <ry...@gmail.com>:
>
> I am trying to set up Solr 8.11.1 on a new server that is running RHEL 8
> along with Java 11 openjdk. I followed all of the steps listed in the Solr
> guide (https://solr.apache.org/guide/8_11/taking-solr-to-production.html)
>
> I pretty much was able to follow the guide with no issue up until the part
> where it checks the progress (where I'm supposed to start solr). That's
> when I try to start Solr and see the error
> "java.lang.IllegalStateException: /opt/solr-8.11.1/server/NONE is not a
> valid keystore"..."Caused by: javax.servlet.UnavailableException: Error
> processing the request. CoreContainer is either not initialized or shutting
> down."
>
> I ran the install script that comes packaged with Solr and which the guide
> says to run. I kept all the default values. So I'm not sure why I'm getting
> this issue. The odd thing is that I was able to install Solr 7.7 with
> similar steps provided on Solr's site and got it to work with no issues. So
> only Solr 8 is not working for me. I've tried a few of the Solr 8.X
> versions, but all gave me the same error and the logs aren't providing any
> other relevant message besides the ones I mentioned above. Any ideas on why
> I'm getting it and how to resolve it? Thank you in advance!
>
>
>
Re: Help! Can't Get Solr 8.11.1 to Work
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi,
Please see in your /etc/default/solr.in <http://solr.in/>.sh file whether you have any SOLR_SSL_* variables defined and if those are correctly set up.
If you cannot find anything, it would help a lot if you would provide the entire log from /var/solr/data/logs/solr.log as well as the entire /etc/default/solr.in <http://solr.in/>.sh file, for debugging.
Jan
> 9. feb. 2022 kl. 22:21 skrev Ryan Work <ry...@gmail.com>:
>
> I am trying to set up Solr 8.11.1 on a new server that is running RHEL 8
> along with Java 11 openjdk. I followed all of the steps listed in the Solr
> guide (https://solr.apache.org/guide/8_11/taking-solr-to-production.html)
>
> I pretty much was able to follow the guide with no issue up until the part
> where it checks the progress (where I'm supposed to start solr). That's
> when I try to start Solr and see the error
> "java.lang.IllegalStateException: /opt/solr-8.11.1/server/NONE is not a
> valid keystore"..."Caused by: javax.servlet.UnavailableException: Error
> processing the request. CoreContainer is either not initialized or shutting
> down."
>
> I ran the install script that comes packaged with Solr and which the guide
> says to run. I kept all the default values. So I'm not sure why I'm getting
> this issue. The odd thing is that I was able to install Solr 7.7 with
> similar steps provided on Solr's site and got it to work with no issues. So
> only Solr 8 is not working for me. I've tried a few of the Solr 8.X
> versions, but all gave me the same error and the logs aren't providing any
> other relevant message besides the ones I mentioned above. Any ideas on why
> I'm getting it and how to resolve it? Thank you in advance!