You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by yaswanth kumar <ya...@gmail.com> on 2021/05/18 02:42:35 UTC
Suggestions required for fixing zookeeper dir config
Solr : 8.2
Zookeeper:3.4
Recently found an issue inside the zoo.cfg where the datadir and logdir are having a dos carriage return characters on the folder path when this is in Linux and that created a folder with ^M characters upon zoo start where we noticed the zookeeper pid file is in. This is creating issues when the servers were rebooted for maintenance purpose and when we have auto start scripts for both zookeeper and solr. Sometimes all the collections were disappeared and also loosing zoo configs that were configured before reboot.
So now this bad config is still in live and we need to correct it carefully by running sed command that replaces these bad carriage returns on these zoo.cfg
Are there a sequence steps that we need to do carefully while we change this config (zoo.cfg) on live severs without any disruption to the cloud??
No of solr severs are 3 and all three got zookeepers on them and we need to fix this bad characters on all 3 solr severs zoo.cfg
Sent from my iPhone
Re: Suggestions required for fixing zookeeper dir config
Posted by Jan Høydahl <ja...@cominvent.com>.
ZK is likely in a corrupt state already, trying to write to a folder with a CR at the end. It may have succeeded doing so, in which case fixing the config will again point it to the correct folder which may then contain an older version of the data (solr's config).
So what I'd do is this
1. Stop all Zookeepers and Solr
2. Decide what dataDir and logDir is the most current / correct
3. If necessary, move/rename these folders to match what you have in your fixed zoo.cfg
4. Start all ZKs - watch them sync up and be healthy
5. Start Solr (if the on-disk state on Solr matches the ZK config then all will come up again)
6. Fix any issues that may have happened in Solr due to the misconfiguration
You may also want to check with the Zookeeper project (https://zookeeper.apache.org/) if they have a fix for this issue (tolerate CR in config file).
Jan
> 18. mai 2021 kl. 15:12 skrev yaswanth kumar <ya...@gmail.com>:
>
> Thanks for explaining about dos2unix stuff, but here my main question is
> that what could be the sequential steps fixing the bad characters on the
> zoo.cfg while the services are running.
>
> Do we need to stop all the solr services on all the servers, stop the
> zookeeper on one server, correct the config , make the zookeeper service up
> and do the same steps on remaining servers as well. Are these steps good
> enough? or am I missing any sequence that could mess up the configs read by
> solr
>
> Thanks,
> Yaswanth
>
> On Tue, May 18, 2021 at 2:45 AM Claude Warren <cl...@xenei.com> wrote:
>
>> A quick fix would be to run the linux command `dos2unix` on the config
>> files. This will ensure that all the end of line (EOL) characters are
>> unix/linux style.
>>
>> A process fix that identifies a Windows tool that will put the Linux EOL
>> character rather than the Windows one might be helpful, but I think you
>> would still need to use the `dos2linux` command just to be safe.
>>
>> You could put `dos2unix` in your startup scripts just to be sure.
>>
>> You could spend some time looking at the code that reads the configuration
>> files and contribute a patch that will remove the Windows EOL and replace
>> it with the Linux EOL. If you do this you might also consider replacing
>> the Mac EOL with Linux EOL in the same code.
>>
>> Claude
>>
>> On Tue, May 18, 2021 at 3:43 AM yaswanth kumar <ya...@gmail.com>
>> wrote:
>>
>>> Solr : 8.2
>>> Zookeeper:3.4
>>>
>>> Recently found an issue inside the zoo.cfg where the datadir and logdir
>>> are having a dos carriage return characters on the folder path when this
>> is
>>> in Linux and that created a folder with ^M characters upon zoo start
>> where
>>> we noticed the zookeeper pid file is in. This is creating issues when the
>>> servers were rebooted for maintenance purpose and when we have auto start
>>> scripts for both zookeeper and solr. Sometimes all the collections were
>>> disappeared and also loosing zoo configs that were configured before
>> reboot.
>>>
>>> So now this bad config is still in live and we need to correct it
>>> carefully by running sed command that replaces these bad carriage returns
>>> on these zoo.cfg
>>>
>>> Are there a sequence steps that we need to do carefully while we change
>>> this config (zoo.cfg) on live severs without any disruption to the
>> cloud??
>>>
>>> No of solr severs are 3 and all three got zookeepers on them and we need
>>> to fix this bad characters on all 3 solr severs zoo.cfg
>>>
>>> Sent from my iPhone
>>
>>
>>
>> --
>> I like: Like Like - The likeliest place on the web
>> <http://like-like.xenei.com>
>> LinkedIn: http://www.linkedin.com/in/claudewarren
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanthcse@gmail.com
Re: Suggestions required for fixing zookeeper dir config
Posted by yaswanth kumar <ya...@gmail.com>.
Thanks for explaining about dos2unix stuff, but here my main question is
that what could be the sequential steps fixing the bad characters on the
zoo.cfg while the services are running.
Do we need to stop all the solr services on all the servers, stop the
zookeeper on one server, correct the config , make the zookeeper service up
and do the same steps on remaining servers as well. Are these steps good
enough? or am I missing any sequence that could mess up the configs read by
solr
Thanks,
Yaswanth
On Tue, May 18, 2021 at 2:45 AM Claude Warren <cl...@xenei.com> wrote:
> A quick fix would be to run the linux command `dos2unix` on the config
> files. This will ensure that all the end of line (EOL) characters are
> unix/linux style.
>
> A process fix that identifies a Windows tool that will put the Linux EOL
> character rather than the Windows one might be helpful, but I think you
> would still need to use the `dos2linux` command just to be safe.
>
> You could put `dos2unix` in your startup scripts just to be sure.
>
> You could spend some time looking at the code that reads the configuration
> files and contribute a patch that will remove the Windows EOL and replace
> it with the Linux EOL. If you do this you might also consider replacing
> the Mac EOL with Linux EOL in the same code.
>
> Claude
>
> On Tue, May 18, 2021 at 3:43 AM yaswanth kumar <ya...@gmail.com>
> wrote:
>
> > Solr : 8.2
> > Zookeeper:3.4
> >
> > Recently found an issue inside the zoo.cfg where the datadir and logdir
> > are having a dos carriage return characters on the folder path when this
> is
> > in Linux and that created a folder with ^M characters upon zoo start
> where
> > we noticed the zookeeper pid file is in. This is creating issues when the
> > servers were rebooted for maintenance purpose and when we have auto start
> > scripts for both zookeeper and solr. Sometimes all the collections were
> > disappeared and also loosing zoo configs that were configured before
> reboot.
> >
> > So now this bad config is still in live and we need to correct it
> > carefully by running sed command that replaces these bad carriage returns
> > on these zoo.cfg
> >
> > Are there a sequence steps that we need to do carefully while we change
> > this config (zoo.cfg) on live severs without any disruption to the
> cloud??
> >
> > No of solr severs are 3 and all three got zookeepers on them and we need
> > to fix this bad characters on all 3 solr severs zoo.cfg
> >
> > Sent from my iPhone
>
>
>
> --
> I like: Like Like - The likeliest place on the web
> <http://like-like.xenei.com>
> LinkedIn: http://www.linkedin.com/in/claudewarren
>
--
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanthcse@gmail.com
Re: Suggestions required for fixing zookeeper dir config
Posted by Claude Warren <cl...@xenei.com>.
A quick fix would be to run the linux command `dos2unix` on the config
files. This will ensure that all the end of line (EOL) characters are
unix/linux style.
A process fix that identifies a Windows tool that will put the Linux EOL
character rather than the Windows one might be helpful, but I think you
would still need to use the `dos2linux` command just to be safe.
You could put `dos2unix` in your startup scripts just to be sure.
You could spend some time looking at the code that reads the configuration
files and contribute a patch that will remove the Windows EOL and replace
it with the Linux EOL. If you do this you might also consider replacing
the Mac EOL with Linux EOL in the same code.
Claude
On Tue, May 18, 2021 at 3:43 AM yaswanth kumar <ya...@gmail.com>
wrote:
> Solr : 8.2
> Zookeeper:3.4
>
> Recently found an issue inside the zoo.cfg where the datadir and logdir
> are having a dos carriage return characters on the folder path when this is
> in Linux and that created a folder with ^M characters upon zoo start where
> we noticed the zookeeper pid file is in. This is creating issues when the
> servers were rebooted for maintenance purpose and when we have auto start
> scripts for both zookeeper and solr. Sometimes all the collections were
> disappeared and also loosing zoo configs that were configured before reboot.
>
> So now this bad config is still in live and we need to correct it
> carefully by running sed command that replaces these bad carriage returns
> on these zoo.cfg
>
> Are there a sequence steps that we need to do carefully while we change
> this config (zoo.cfg) on live severs without any disruption to the cloud??
>
> No of solr severs are 3 and all three got zookeepers on them and we need
> to fix this bad characters on all 3 solr severs zoo.cfg
>
> Sent from my iPhone
--
I like: Like Like - The likeliest place on the web
<http://like-like.xenei.com>
LinkedIn: http://www.linkedin.com/in/claudewarren