You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Eliot Kimber <el...@servicenow.com.INVALID> on 2025/01/28 13:43:29 UTC
Solo server restarting every 90 seconds for no clear reason
I have installed Solr on RHEL 9 and it is working fine except that it restarts every 90 seconds. My installation is about as vanilla as it can be with minimal changes to the default configuration. It is handling document posts and search requests fine.
The server is exposed through a reverse proxy using Apache httpd, which is also serving a statically-generate web site.
Based on what I’ve found online, which wasn’t much for this particular behavior, I’ve searched for any cron job that might be restarting it (there is not one) or any other reason.
I haven’t been able to find anything and I’m not sure where else to look.
It’s not a memory issue—the server doesn’t come close to using the memory allocated to it and I increased it to 2GB from the default just to be sure. I don’t see any errors in the solr.log.
Any idea what I can look for that might be causing the restart or what additional information I can provide here that would help?
Thanks,
Eliot
_____________________________________________
Eliot Kimber
Sr. Staff Content Engineer
O: 512 554 9368
servicenow
servicenow.com<https://www.servicenow.com>
LinkedIn<https://www.linkedin.com/company/servicenow> | X<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Instagram<https://www.instagram.com/servicenow>
Re: Solo server restarting every 90 seconds for no clear reason
Posted by Bernd Fehling <be...@uni-bielefeld.de>.
Make sure that you have not any job / cron running for wiping any of your temp directories.
Regards
Bernd
Am 28.01.25 um 14:43 schrieb Eliot Kimber:
> I have installed Solr on RHEL 9 and it is working fine except that it restarts every 90 seconds. My installation is about as vanilla as it can be with minimal changes to the default configuration. It is handling document posts and search requests fine.
>
> The server is exposed through a reverse proxy using Apache httpd, which is also serving a statically-generate web site.
>
> Based on what I’ve found online, which wasn’t much for this particular behavior, I’ve searched for any cron job that might be restarting it (there is not one) or any other reason.
>
> I haven’t been able to find anything and I’m not sure where else to look.
>
> It’s not a memory issue—the server doesn’t come close to using the memory allocated to it and I increased it to 2GB from the default just to be sure. I don’t see any errors in the solr.log.
>
> Any idea what I can look for that might be causing the restart or what additional information I can provide here that would help?
>
> Thanks,
>
> Eliot
> _____________________________________________
> Eliot Kimber
> Sr. Staff Content Engineer
> O: 512 554 9368
>
> servicenow
>
> servicenow.com<https://www.servicenow.com>
> LinkedIn<https://www.linkedin.com/company/servicenow> | X<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Instagram<https://www.instagram.com/servicenow>
>
AW: Solo server restarting every 90 seconds for no clear reason
Posted by Andreas Mock <an...@drumedar.de>.
Thank you for comming back with your findings and solution...
> -----Ursprüngliche Nachricht-----
> Von: Eliot Kimber <el...@servicenow.com.INVALID>
> Gesendet: Dienstag, 28. Januar 2025 23:25
> An: users@solr.apache.org
> Betreff: Re: Solo server restarting every 90 seconds for no clear reason
>
> This solr.service file seems to be working as intended (no unwanted
> restarts):
>
> [Unit]
> Description=Apache SOLR
>
> [Service]
> Type=forking
> User=solr
> Environment=SOLR_INCLUDE=/etc/default/solr.in.sh
> ExecStart=/opt/solr/bin/solr start
> ExecStop=/opt/solr/bin/solr stop
> Restart=on-failure
> LimitNOFILE=65000
> LimitNPROC=65000
> TimeoutSec=180s
>
> [Install]
> WantedBy=multi-user.target
>
> Cheers,
>
> Eliot
> _____________________________________________
> Eliot Kimber
> Sr. Staff Content Engineer
> O: 512 554 9368
>
> servicenow
>
> servicenow.com<https://www.servicenow.com>
> LinkedIn<https://www.linkedin.com/company/servicenow> |
> X<https://twitter.com/servicenow> |
> YouTube<https://www.youtube.com/user/servicenowinc> |
> Instagram<https://www.instagram.com/servicenow>
>
> From: Eliot Kimber <el...@servicenow.com.INVALID>
> Date: Tuesday, January 28, 2025 at 4:13 PM
> To: users@solr.apache.org <us...@solr.apache.org>
> Subject: Re: Solo server restarting every 90 seconds for no clear reason
> [External Email]
>
> I found my issue: it was own fault: I had created a systemctl service for
> solr and that was causing the issue (see
> https://gist.github.com/hammady/3d7b5964c7b0f90997865ebef40bf5e1<https://g
> ist.github.com/hammady/3d7b5964c7b0f90997865ebef40bf5e1>)
>
> The systemctl process starts Solr but then immediately shuts it down,
> which apparently takes 90 seconds, and then it helpfully restarts it.
>
> Starting Solr using the solr command works fine and it stays up.
>
> I was setting this server up very quickly without really knowing what I
> was doing, so there you go. I had forgotten that’d done this setup
> manually rather than through some Solr-supported process.
>
> Sorry to have bothered the group but maybe somebody else will find this
> discussion useful in the future.
>
> Cheers,
>
> Eliot
>
> _____________________________________________
> Eliot Kimber
> Sr. Staff Content Engineer
> O: 512 554 9368
>
> servicenow
>
> servicenow.com<https://www.servicenow.com<https://www.servicenow.com>>
> LinkedIn<https://www.linkedin.com/company/servicenow<https://www.linkedin.
> com/company/servicenow>> |
> X<https://twitter.com/servicenow<https://twitter.com/servicenow>> |
> YouTube<https://www.youtube.com/user/servicenowinc<https://www.youtube.com
> /user/servicenowinc>> |
> Instagram<https://www.instagram.com/servicenow<https://www.instagram.com/s
> ervicenow>>
>
> From: Eliot Kimber <el...@servicenow.com.INVALID>
> Date: Tuesday, January 28, 2025 at 12:53 PM
> To: users@solr.apache.org <us...@solr.apache.org>
> Subject: Re: Solo server restarting every 90 seconds for no clear reason
> [External Email]
>
>
> Solr 9.7.0
>
> I downloaded it and manually installed.
>
> I have full control of this server and thus the cron jobs, and there are
> no cron jobs that should be interfering with Solr in any way and certainly
> none that run ever 90 seconds (my cron jobs run about once an hour on this
> server).
>
> I found a Solr-specific directory under /tmp (jetty-127_0_0_1-8983-webapp-
> _solr-any-6912854890539458980) and it changes at the same 90 second
> interval, but I’m guessing that’s because the server restarts, not because
> something is interfering with it.
>
> All the directories are the default locations (i.e. /var/solr/…) and I
> haven’t done anything like simlink the /data directory to a different file
> system (my /var file system is small but still has plenty of space at the
> moment).
>
> This is a corporate managed VM so it’s possible that it’s being subjected
> to some regular command that has the effect of triggering Solr to restart.
>
> Can someone confirm that there’s no aspect of a normal Solr configuration
> (and certainly nothing in the default configuration) that would cause this
> sort of periodic restart?
>
> Since I didn’t find anything similar to this issue in my searching other
> than an out-of-memory condition triggering it, I’m thinking it must be
> something specific to my server and non-obvious.
>
> Thanks,
>
> Eliot
> _____________________________________________
> Eliot Kimber
> Sr. Staff Content Engineer
> O: 512 554 9368
>
> servicenow
>
> servicenow.com<https://www.servicenow.com<https://www.servicenow.com><http
> s://www.servicenow.com<https://www.servicenow.com>>>
> LinkedIn<https://www.linkedin.com/company/servicenow<https://www.linkedin.
> com/company/servicenow><https://www.linkedin.com/company/servicenow<https:
> //www.linkedin.com/company/servicenow>>> |
> X<https://twitter.com/servicenow<https://twitter.com/servicenow><https://t
> witter.com/servicenow<https://twitter.com/servicenow>>> |
> YouTube<https://www.youtube.com/user/servicenowinc<https://www.youtube.com
> /user/servicenowinc><https://www.youtube.com/user/servicenowinc<https://ww
> w.youtube.com/user/servicenowinc>>> |
> Instagram<https://www.instagram.com/servicenow<https://www.instagram.com/s
> ervicenow><https://www.instagram.com/servicenow<https://www.instagram.com/
> servicenow>>>
>
> From: Andreas Mock <an...@drumedar.de>
> Date: Tuesday, January 28, 2025 at 9:46 AM
> To: users@solr.apache.org <us...@solr.apache.org>
> Subject: AW: Solo server restarting every 90 seconds for no clear reason
> [External Email]
>
> > -----Ursprüngliche Nachricht-----
> > Von: Eliot Kimber <el...@servicenow.com.INVALID>
> > Gesendet: Dienstag, 28. Januar 2025 14:43
> > An: users@solr.apache.org
> > Betreff: Solo server restarting every 90 seconds for no clear reason
> >
>
>
> > I have installed Solr on RHEL 9 and it is working fine except that it
> > restarts every 90 seconds.
>
> Which version?
> Which kind of installation did you use? (dnf-driven, manual with tgz)
>
> Best regards
> Andreas
Re: Solo server restarting every 90 seconds for no clear reason
Posted by Eliot Kimber <el...@servicenow.com.INVALID>.
This solr.service file seems to be working as intended (no unwanted restarts):
[Unit]
Description=Apache SOLR
[Service]
Type=forking
User=solr
Environment=SOLR_INCLUDE=/etc/default/solr.in.sh
ExecStart=/opt/solr/bin/solr start
ExecStop=/opt/solr/bin/solr stop
Restart=on-failure
LimitNOFILE=65000
LimitNPROC=65000
TimeoutSec=180s
[Install]
WantedBy=multi-user.target
Cheers,
Eliot
_____________________________________________
Eliot Kimber
Sr. Staff Content Engineer
O: 512 554 9368
servicenow
servicenow.com<https://www.servicenow.com>
LinkedIn<https://www.linkedin.com/company/servicenow> | X<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Instagram<https://www.instagram.com/servicenow>
From: Eliot Kimber <el...@servicenow.com.INVALID>
Date: Tuesday, January 28, 2025 at 4:13 PM
To: users@solr.apache.org <us...@solr.apache.org>
Subject: Re: Solo server restarting every 90 seconds for no clear reason
[External Email]
I found my issue: it was own fault: I had created a systemctl service for solr and that was causing the issue (see https://gist.github.com/hammady/3d7b5964c7b0f90997865ebef40bf5e1<https://gist.github.com/hammady/3d7b5964c7b0f90997865ebef40bf5e1>)
The systemctl process starts Solr but then immediately shuts it down, which apparently takes 90 seconds, and then it helpfully restarts it.
Starting Solr using the solr command works fine and it stays up.
I was setting this server up very quickly without really knowing what I was doing, so there you go. I had forgotten that’d done this setup manually rather than through some Solr-supported process.
Sorry to have bothered the group but maybe somebody else will find this discussion useful in the future.
Cheers,
Eliot
_____________________________________________
Eliot Kimber
Sr. Staff Content Engineer
O: 512 554 9368
servicenow
servicenow.com<https://www.servicenow.com<https://www.servicenow.com>>
LinkedIn<https://www.linkedin.com/company/servicenow<https://www.linkedin.com/company/servicenow>> | X<https://twitter.com/servicenow<https://twitter.com/servicenow>> | YouTube<https://www.youtube.com/user/servicenowinc<https://www.youtube.com/user/servicenowinc>> | Instagram<https://www.instagram.com/servicenow<https://www.instagram.com/servicenow>>
From: Eliot Kimber <el...@servicenow.com.INVALID>
Date: Tuesday, January 28, 2025 at 12:53 PM
To: users@solr.apache.org <us...@solr.apache.org>
Subject: Re: Solo server restarting every 90 seconds for no clear reason
[External Email]
Solr 9.7.0
I downloaded it and manually installed.
I have full control of this server and thus the cron jobs, and there are no cron jobs that should be interfering with Solr in any way and certainly none that run ever 90 seconds (my cron jobs run about once an hour on this server).
I found a Solr-specific directory under /tmp (jetty-127_0_0_1-8983-webapp-_solr-any-6912854890539458980) and it changes at the same 90 second interval, but I’m guessing that’s because the server restarts, not because something is interfering with it.
All the directories are the default locations (i.e. /var/solr/…) and I haven’t done anything like simlink the /data directory to a different file system (my /var file system is small but still has plenty of space at the moment).
This is a corporate managed VM so it’s possible that it’s being subjected to some regular command that has the effect of triggering Solr to restart.
Can someone confirm that there’s no aspect of a normal Solr configuration (and certainly nothing in the default configuration) that would cause this sort of periodic restart?
Since I didn’t find anything similar to this issue in my searching other than an out-of-memory condition triggering it, I’m thinking it must be something specific to my server and non-obvious.
Thanks,
Eliot
_____________________________________________
Eliot Kimber
Sr. Staff Content Engineer
O: 512 554 9368
servicenow
servicenow.com<https://www.servicenow.com<https://www.servicenow.com><https://www.servicenow.com<https://www.servicenow.com>>>
LinkedIn<https://www.linkedin.com/company/servicenow<https://www.linkedin.com/company/servicenow><https://www.linkedin.com/company/servicenow<https://www.linkedin.com/company/servicenow>>> | X<https://twitter.com/servicenow<https://twitter.com/servicenow><https://twitter.com/servicenow<https://twitter.com/servicenow>>> | YouTube<https://www.youtube.com/user/servicenowinc<https://www.youtube.com/user/servicenowinc><https://www.youtube.com/user/servicenowinc<https://www.youtube.com/user/servicenowinc>>> | Instagram<https://www.instagram.com/servicenow<https://www.instagram.com/servicenow><https://www.instagram.com/servicenow<https://www.instagram.com/servicenow>>>
From: Andreas Mock <an...@drumedar.de>
Date: Tuesday, January 28, 2025 at 9:46 AM
To: users@solr.apache.org <us...@solr.apache.org>
Subject: AW: Solo server restarting every 90 seconds for no clear reason
[External Email]
> -----Ursprüngliche Nachricht-----
> Von: Eliot Kimber <el...@servicenow.com.INVALID>
> Gesendet: Dienstag, 28. Januar 2025 14:43
> An: users@solr.apache.org
> Betreff: Solo server restarting every 90 seconds for no clear reason
>
> I have installed Solr on RHEL 9 and it is working fine except that it
> restarts every 90 seconds.
Which version?
Which kind of installation did you use? (dnf-driven, manual with tgz)
Best regards
Andreas
Re: Solo server restarting every 90 seconds for no clear reason
Posted by Eliot Kimber <el...@servicenow.com.INVALID>.
I found my issue: it was own fault: I had created a systemctl service for solr and that was causing the issue (see https://gist.github.com/hammady/3d7b5964c7b0f90997865ebef40bf5e1)
The systemctl process starts Solr but then immediately shuts it down, which apparently takes 90 seconds, and then it helpfully restarts it.
Starting Solr using the solr command works fine and it stays up.
I was setting this server up very quickly without really knowing what I was doing, so there you go. I had forgotten that’d done this setup manually rather than through some Solr-supported process.
Sorry to have bothered the group but maybe somebody else will find this discussion useful in the future.
Cheers,
Eliot
_____________________________________________
Eliot Kimber
Sr. Staff Content Engineer
O: 512 554 9368
servicenow
servicenow.com<https://www.servicenow.com>
LinkedIn<https://www.linkedin.com/company/servicenow> | X<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Instagram<https://www.instagram.com/servicenow>
From: Eliot Kimber <el...@servicenow.com.INVALID>
Date: Tuesday, January 28, 2025 at 12:53 PM
To: users@solr.apache.org <us...@solr.apache.org>
Subject: Re: Solo server restarting every 90 seconds for no clear reason
[External Email]
Solr 9.7.0
I downloaded it and manually installed.
I have full control of this server and thus the cron jobs, and there are no cron jobs that should be interfering with Solr in any way and certainly none that run ever 90 seconds (my cron jobs run about once an hour on this server).
I found a Solr-specific directory under /tmp (jetty-127_0_0_1-8983-webapp-_solr-any-6912854890539458980) and it changes at the same 90 second interval, but I’m guessing that’s because the server restarts, not because something is interfering with it.
All the directories are the default locations (i.e. /var/solr/…) and I haven’t done anything like simlink the /data directory to a different file system (my /var file system is small but still has plenty of space at the moment).
This is a corporate managed VM so it’s possible that it’s being subjected to some regular command that has the effect of triggering Solr to restart.
Can someone confirm that there’s no aspect of a normal Solr configuration (and certainly nothing in the default configuration) that would cause this sort of periodic restart?
Since I didn’t find anything similar to this issue in my searching other than an out-of-memory condition triggering it, I’m thinking it must be something specific to my server and non-obvious.
Thanks,
Eliot
_____________________________________________
Eliot Kimber
Sr. Staff Content Engineer
O: 512 554 9368
servicenow
servicenow.com<https://www.servicenow.com<https://www.servicenow.com>>
LinkedIn<https://www.linkedin.com/company/servicenow<https://www.linkedin.com/company/servicenow>> | X<https://twitter.com/servicenow<https://twitter.com/servicenow>> | YouTube<https://www.youtube.com/user/servicenowinc<https://www.youtube.com/user/servicenowinc>> | Instagram<https://www.instagram.com/servicenow<https://www.instagram.com/servicenow>>
From: Andreas Mock <an...@drumedar.de>
Date: Tuesday, January 28, 2025 at 9:46 AM
To: users@solr.apache.org <us...@solr.apache.org>
Subject: AW: Solo server restarting every 90 seconds for no clear reason
[External Email]
> -----Ursprüngliche Nachricht-----
> Von: Eliot Kimber <el...@servicenow.com.INVALID>
> Gesendet: Dienstag, 28. Januar 2025 14:43
> An: users@solr.apache.org
> Betreff: Solo server restarting every 90 seconds for no clear reason
>
> I have installed Solr on RHEL 9 and it is working fine except that it
> restarts every 90 seconds.
Which version?
Which kind of installation did you use? (dnf-driven, manual with tgz)
Best regards
Andreas
Re: Solo server restarting every 90 seconds for no clear reason
Posted by Eliot Kimber <el...@servicenow.com.INVALID>.
Solr 9.7.0
I downloaded it and manually installed.
I have full control of this server and thus the cron jobs, and there are no cron jobs that should be interfering with Solr in any way and certainly none that run ever 90 seconds (my cron jobs run about once an hour on this server).
I found a Solr-specific directory under /tmp (jetty-127_0_0_1-8983-webapp-_solr-any-6912854890539458980) and it changes at the same 90 second interval, but I’m guessing that’s because the server restarts, not because something is interfering with it.
All the directories are the default locations (i.e. /var/solr/…) and I haven’t done anything like simlink the /data directory to a different file system (my /var file system is small but still has plenty of space at the moment).
This is a corporate managed VM so it’s possible that it’s being subjected to some regular command that has the effect of triggering Solr to restart.
Can someone confirm that there’s no aspect of a normal Solr configuration (and certainly nothing in the default configuration) that would cause this sort of periodic restart?
Since I didn’t find anything similar to this issue in my searching other than an out-of-memory condition triggering it, I’m thinking it must be something specific to my server and non-obvious.
Thanks,
Eliot
_____________________________________________
Eliot Kimber
Sr. Staff Content Engineer
O: 512 554 9368
servicenow
servicenow.com<https://www.servicenow.com>
LinkedIn<https://www.linkedin.com/company/servicenow> | X<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Instagram<https://www.instagram.com/servicenow>
From: Andreas Mock <an...@drumedar.de>
Date: Tuesday, January 28, 2025 at 9:46 AM
To: users@solr.apache.org <us...@solr.apache.org>
Subject: AW: Solo server restarting every 90 seconds for no clear reason
[External Email]
> -----Ursprüngliche Nachricht-----
> Von: Eliot Kimber <el...@servicenow.com.INVALID>
> Gesendet: Dienstag, 28. Januar 2025 14:43
> An: users@solr.apache.org
> Betreff: Solo server restarting every 90 seconds for no clear reason
>
> I have installed Solr on RHEL 9 and it is working fine except that it
> restarts every 90 seconds.
Which version?
Which kind of installation did you use? (dnf-driven, manual with tgz)
Best regards
Andreas
AW: Solo server restarting every 90 seconds for no clear reason
Posted by Andreas Mock <an...@drumedar.de>.
> -----Ursprüngliche Nachricht-----
> Von: Eliot Kimber <el...@servicenow.com.INVALID>
> Gesendet: Dienstag, 28. Januar 2025 14:43
> An: users@solr.apache.org
> Betreff: Solo server restarting every 90 seconds for no clear reason
>
> I have installed Solr on RHEL 9 and it is working fine except that it
> restarts every 90 seconds.
Which version?
Which kind of installation did you use? (dnf-driven, manual with tgz)
Best regards
Andreas