Can solrcloud be running on a read-only filesystem?

[Wudong Liu <wu...@gmail.com>]

Hi All:

We have a normal build/stage -> prod settings for our production pipeline.
And we would build solr index in the build environment and then the index
is copied to the prod environment.

The solrcloud in prod seems working fine when the file system backing it is
writable. However, we see many errors when the file system is readonly.
Many exceptions are thrown regarding the tlog file cannot be open for write
when the solr nodes are restarted with the new data; some of the nodes
eventually are stuck in the recovering phase and never able to go back
online in the cloud.

Just wondering is anyone has any experience on Solrcloud running in
readonly file system? Is it possible at all?

Regards,
Wudong

Re: Can solrcloud be running on a read-only filesystem?

[Erick Erickson <er...@gmail.com>]

Mike:

That's one possibility. What I'm really asking for is to be sure that
there's a good reason (yours is one).

It's just that I've spent too much time in my life trying to get
something to work only to discover that it has marginal utility so I
like to ask "is this important enough to take time away from other
work you could be doing?" ;). If the answer is "yes", then.....

The other possibility is to have multiple Solr instances reading one
Solr index. Which is possible but risky. Having the filesystem R/O
would prevent multiple Solr's writing to it of course. If that's the
case we get to argue whether it's worth it or not of course ;)...

Best,
Erick

On Fri, Jun 2, 2017 at 8:47 AM, Mike Drob <ma...@cloudera.com> wrote:
> To throw out one possibility, a read only file systems has no (low?)
> possibility of corruption. If you have a static index then you shouldn't
> need to be doing any recovery. Would still need to run ZK with RW
> filesystem, but maaaaybe Solr could work?
>
> On Fri, Jun 2, 2017 at 10:15 AM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> As Susheel says, this is iffy, very iffy. You can disable tlogs
>> entirely through solrconfig.xml, you can _probably_
>> disable all of the Solr logging.
>>
>> You'd also have to _not_ run in SolrCloud. You say
>> "some of the nodes eventually are stuck in the recovering phase"
>> SolrCloud tries very hard to keep all of the replicas in sync.
>> To do this it _must_ be able to copy from the leader to the follower.
>> If it ever has to sync with the leader, it'll be stuck in recovery
>> as you can see.
>>
>> You could spend a lot of time trying to make this work, but
>> you haven't stated _why_ you want to. Perhaps there are
>> other ways to get the functionality you want.
>>
>> Best,
>> Erick
>>
>> On Fri, Jun 2, 2017 at 5:05 AM, Susheel Kumar <su...@gmail.com>
>> wrote:
>> > I doubt it can run in readonly file system.  Even though there is no
>> > ingestion etc.  Solr still needs to write to logs/tlogs for synching /
>> > recovering etc....
>> >
>> > Thnx
>> >
>> > On Fri, Jun 2, 2017 at 6:56 AM, Wudong Liu <wu...@gmail.com> wrote:
>> >
>> >> Hi All:
>> >>
>> >> We have a normal build/stage -> prod settings for our production
>> pipeline.
>> >> And we would build solr index in the build environment and then the
>> index
>> >> is copied to the prod environment.
>> >>
>> >> The solrcloud in prod seems working fine when the file system backing
>> it is
>> >> writable. However, we see many errors when the file system is readonly.
>> >> Many exceptions are thrown regarding the tlog file cannot be open for
>> write
>> >> when the solr nodes are restarted with the new data; some of the nodes
>> >> eventually are stuck in the recovering phase and never able to go back
>> >> online in the cloud.
>> >>
>> >> Just wondering is anyone has any experience on Solrcloud running in
>> >> readonly file system? Is it possible at all?
>> >>
>> >> Regards,
>> >> Wudong
>> >>
>>

Re: Can solrcloud be running on a read-only filesystem?

[Mike Drob <ma...@cloudera.com>]

To throw out one possibility, a read only file systems has no (low?)
possibility of corruption. If you have a static index then you shouldn't
need to be doing any recovery. Would still need to run ZK with RW
filesystem, but maaaaybe Solr could work?

On Fri, Jun 2, 2017 at 10:15 AM, Erick Erickson <er...@gmail.com>
wrote:

> As Susheel says, this is iffy, very iffy. You can disable tlogs
> entirely through solrconfig.xml, you can _probably_
> disable all of the Solr logging.
>
> You'd also have to _not_ run in SolrCloud. You say
> "some of the nodes eventually are stuck in the recovering phase"
> SolrCloud tries very hard to keep all of the replicas in sync.
> To do this it _must_ be able to copy from the leader to the follower.
> If it ever has to sync with the leader, it'll be stuck in recovery
> as you can see.
>
> You could spend a lot of time trying to make this work, but
> you haven't stated _why_ you want to. Perhaps there are
> other ways to get the functionality you want.
>
> Best,
> Erick
>
> On Fri, Jun 2, 2017 at 5:05 AM, Susheel Kumar <su...@gmail.com>
> wrote:
> > I doubt it can run in readonly file system.  Even though there is no
> > ingestion etc.  Solr still needs to write to logs/tlogs for synching /
> > recovering etc....
> >
> > Thnx
> >
> > On Fri, Jun 2, 2017 at 6:56 AM, Wudong Liu <wu...@gmail.com> wrote:
> >
> >> Hi All:
> >>
> >> We have a normal build/stage -> prod settings for our production
> pipeline.
> >> And we would build solr index in the build environment and then the
> index
> >> is copied to the prod environment.
> >>
> >> The solrcloud in prod seems working fine when the file system backing
> it is
> >> writable. However, we see many errors when the file system is readonly.
> >> Many exceptions are thrown regarding the tlog file cannot be open for
> write
> >> when the solr nodes are restarted with the new data; some of the nodes
> >> eventually are stuck in the recovering phase and never able to go back
> >> online in the cloud.
> >>
> >> Just wondering is anyone has any experience on Solrcloud running in
> >> readonly file system? Is it possible at all?
> >>
> >> Regards,
> >> Wudong
> >>
>

Re: Can solrcloud be running on a read-only filesystem?

[Erick Erickson <er...@gmail.com>]

As Susheel says, this is iffy, very iffy. You can disable tlogs
entirely through solrconfig.xml, you can _probably_
disable all of the Solr logging.

You'd also have to _not_ run in SolrCloud. You say
"some of the nodes eventually are stuck in the recovering phase"
SolrCloud tries very hard to keep all of the replicas in sync.
To do this it _must_ be able to copy from the leader to the follower.
If it ever has to sync with the leader, it'll be stuck in recovery
as you can see.

You could spend a lot of time trying to make this work, but
you haven't stated _why_ you want to. Perhaps there are
other ways to get the functionality you want.

Best,
Erick

On Fri, Jun 2, 2017 at 5:05 AM, Susheel Kumar <su...@gmail.com> wrote:
> I doubt it can run in readonly file system.  Even though there is no
> ingestion etc.  Solr still needs to write to logs/tlogs for synching /
> recovering etc....
>
> Thnx
>
> On Fri, Jun 2, 2017 at 6:56 AM, Wudong Liu <wu...@gmail.com> wrote:
>
>> Hi All:
>>
>> We have a normal build/stage -> prod settings for our production pipeline.
>> And we would build solr index in the build environment and then the index
>> is copied to the prod environment.
>>
>> The solrcloud in prod seems working fine when the file system backing it is
>> writable. However, we see many errors when the file system is readonly.
>> Many exceptions are thrown regarding the tlog file cannot be open for write
>> when the solr nodes are restarted with the new data; some of the nodes
>> eventually are stuck in the recovering phase and never able to go back
>> online in the cloud.
>>
>> Just wondering is anyone has any experience on Solrcloud running in
>> readonly file system? Is it possible at all?
>>
>> Regards,
>> Wudong
>>

Re: Can solrcloud be running on a read-only filesystem?

[Susheel Kumar <su...@gmail.com>]

I doubt it can run in readonly file system.  Even though there is no
ingestion etc.  Solr still needs to write to logs/tlogs for synching /
recovering etc....

Thnx

On Fri, Jun 2, 2017 at 6:56 AM, Wudong Liu <wu...@gmail.com> wrote:

> Hi All:
>
> We have a normal build/stage -> prod settings for our production pipeline.
> And we would build solr index in the build environment and then the index
> is copied to the prod environment.
>
> The solrcloud in prod seems working fine when the file system backing it is
> writable. However, we see many errors when the file system is readonly.
> Many exceptions are thrown regarding the tlog file cannot be open for write
> when the solr nodes are restarted with the new data; some of the nodes
> eventually are stuck in the recovering phase and never able to go back
> online in the cloud.
>
> Just wondering is anyone has any experience on Solrcloud running in
> readonly file system? Is it possible at all?
>
> Regards,
> Wudong
>