lazily-loaded cores and SolrCloud

[Erick Erickson <er...@gmail.com>]

There was a question on the user's list today about making lazily-loaded
(aka transient) cores work with SolrCloud where I basically punted and said
"not designed with that in mind". I've kind of avoided thinking about this
as the use-case; the transient code wasn't written with SolrCloud in mind.

But what is the general reaction to that pairing? Mostly I'm looking for
feedback at the level of "no way that could work without invasive changes
to SolrCloud, don't even go there" or "sure, just allow ZK to get a list of
all cores and it'll be fine, the user is responsible for the quirks
though". Some questions that come to my mind:

> Is a core that's not loaded be considered "live" by ZK? Would simply
returning a list of all cores (both loaded and not loaded) be sufficient
for ZK? (this list is already available so the admin UI can list all cores).

> Does SolrCloud distributed update processing go through (or could be made
to go through) the path that autoloads a core?

> Ditto for querying. I suspect the answer to both is that it'll "just
happen".

> Would the idea of waiting for all the cores to load on all the nodes for
an update be totally unacceptable? We already have the distributed deadlock
potential, this seems to make that more likely by lengthening out the time
the semaphore in question is held.

> Would re-synching/leader election be an absolute nightmare? I can imagine
that if all the cores for a particular shard weren't loaded at startup,
there'd be a terrible time waiting for leader election for instance.

> Stuff I haven't thought of

Mostly I'm trying to get a "sense of the community" here about whether
supporting transient cores in SolrCloud mode would be something that would
be easy/do-able/really_hard/totally_unacceptable.

Thanks,
Erick

Re: lazily-loaded cores and SolrCloud

[Erick Erickson <er...@gmail.com>]

See SOLR-5146, it's an "investigative" JIRA.

I won't be able to look at this in the near future, so anyone who wants
to feel free to take it!

Erick


On Tue, Aug 13, 2013 at 8:13 PM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:

> +1
> On 14 Aug 2013 05:39, "Noble Paul നോബിള്‍ नोब्ळ्" <no...@gmail.com>
> wrote:
>
>> It is worth doing it. A lot of people will need it  . Let's open an issue
>> and come up with a design for the same .
>> On 14 Aug 2013 02:16, "Yonik Seeley" <yo...@lucidworks.com> wrote:
>>
>>> At a high level, I think the idea is fine (and I've seen a number of
>>> people that wanted it).
>>> The question is more around one of implementation... would it make a
>>> mess of things or not.
>>> The answer to that I think is probably mostly related to issues around
>>> how zookeeper is currently handled.
>>> I don't see any issues with other things like spinning up a core when
>>> a request comes in for it.
>>>
>>> -Yonik
>>> http://lucidworks.com
>>>
>>> On Tue, Aug 13, 2013 at 4:26 PM, Erick Erickson <er...@gmail.com>
>>> wrote:
>>> > There was a question on the user's list today about making
>>> lazily-loaded
>>> > (aka transient) cores work with SolrCloud where I basically punted and
>>> said
>>> > "not designed with that in mind". I've kind of avoided thinking about
>>> this
>>> > as the use-case; the transient code wasn't written with SolrCloud in
>>> mind.
>>> >
>>> > But what is the general reaction to that pairing? Mostly I'm looking
>>> for
>>> > feedback at the level of "no way that could work without invasive
>>> changes to
>>> > SolrCloud, don't even go there" or "sure, just allow ZK to get a list
>>> of all
>>> > cores and it'll be fine, the user is responsible for the quirks
>>> though".
>>> > Some questions that come to my mind:
>>> >
>>> >> Is a core that's not loaded be considered "live" by ZK? Would simply
>>> >> returning a list of all cores (both loaded and not loaded) be
>>> sufficient for
>>> >> ZK? (this list is already available so the admin UI can list all
>>> cores).
>>> >
>>> >> Does SolrCloud distributed update processing go through (or could be
>>> made
>>> >> to go through) the path that autoloads a core?
>>> >
>>> >> Ditto for querying. I suspect the answer to both is that it'll "just
>>> >> happen".
>>> >
>>> >> Would the idea of waiting for all the cores to load on all the nodes
>>> for
>>> >> an update be totally unacceptable? We already have the distributed
>>> deadlock
>>> >> potential, this seems to make that more likely by lengthening out the
>>> time
>>> >> the semaphore in question is held.
>>> >
>>> >> Would re-synching/leader election be an absolute nightmare? I can
>>> imagine
>>> >> that if all the cores for a particular shard weren't loaded at
>>> startup,
>>> >> there'd be a terrible time waiting for leader election for instance.
>>> >
>>> >> Stuff I haven't thought of
>>> >
>>> > Mostly I'm trying to get a "sense of the community" here about whether
>>> > supporting transient cores in SolrCloud mode would be something that
>>> would
>>> > be easy/do-able/really_hard/totally_unacceptable.
>>> >
>>> > Thanks,
>>> > Erick
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>>

Re: lazily-loaded cores and SolrCloud

[Shalin Shekhar Mangar <sh...@gmail.com>]

+1
On 14 Aug 2013 05:39, "Noble Paul നോബിള്‍ नोब्ळ्" <no...@gmail.com>
wrote:

> It is worth doing it. A lot of people will need it  . Let's open an issue
> and come up with a design for the same .
> On 14 Aug 2013 02:16, "Yonik Seeley" <yo...@lucidworks.com> wrote:
>
>> At a high level, I think the idea is fine (and I've seen a number of
>> people that wanted it).
>> The question is more around one of implementation... would it make a
>> mess of things or not.
>> The answer to that I think is probably mostly related to issues around
>> how zookeeper is currently handled.
>> I don't see any issues with other things like spinning up a core when
>> a request comes in for it.
>>
>> -Yonik
>> http://lucidworks.com
>>
>> On Tue, Aug 13, 2013 at 4:26 PM, Erick Erickson <er...@gmail.com>
>> wrote:
>> > There was a question on the user's list today about making lazily-loaded
>> > (aka transient) cores work with SolrCloud where I basically punted and
>> said
>> > "not designed with that in mind". I've kind of avoided thinking about
>> this
>> > as the use-case; the transient code wasn't written with SolrCloud in
>> mind.
>> >
>> > But what is the general reaction to that pairing? Mostly I'm looking for
>> > feedback at the level of "no way that could work without invasive
>> changes to
>> > SolrCloud, don't even go there" or "sure, just allow ZK to get a list
>> of all
>> > cores and it'll be fine, the user is responsible for the quirks though".
>> > Some questions that come to my mind:
>> >
>> >> Is a core that's not loaded be considered "live" by ZK? Would simply
>> >> returning a list of all cores (both loaded and not loaded) be
>> sufficient for
>> >> ZK? (this list is already available so the admin UI can list all
>> cores).
>> >
>> >> Does SolrCloud distributed update processing go through (or could be
>> made
>> >> to go through) the path that autoloads a core?
>> >
>> >> Ditto for querying. I suspect the answer to both is that it'll "just
>> >> happen".
>> >
>> >> Would the idea of waiting for all the cores to load on all the nodes
>> for
>> >> an update be totally unacceptable? We already have the distributed
>> deadlock
>> >> potential, this seems to make that more likely by lengthening out the
>> time
>> >> the semaphore in question is held.
>> >
>> >> Would re-synching/leader election be an absolute nightmare? I can
>> imagine
>> >> that if all the cores for a particular shard weren't loaded at startup,
>> >> there'd be a terrible time waiting for leader election for instance.
>> >
>> >> Stuff I haven't thought of
>> >
>> > Mostly I'm trying to get a "sense of the community" here about whether
>> > supporting transient cores in SolrCloud mode would be something that
>> would
>> > be easy/do-able/really_hard/totally_unacceptable.
>> >
>> > Thanks,
>> > Erick
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>

Re: lazily-loaded cores and SolrCloud

[Noble Paul നോബിള്‍ नोब्ळ् <no...@gmail.com>]

It is worth doing it. A lot of people will need it  . Let's open an issue
and come up with a design for the same .
On 14 Aug 2013 02:16, "Yonik Seeley" <yo...@lucidworks.com> wrote:

> At a high level, I think the idea is fine (and I've seen a number of
> people that wanted it).
> The question is more around one of implementation... would it make a
> mess of things or not.
> The answer to that I think is probably mostly related to issues around
> how zookeeper is currently handled.
> I don't see any issues with other things like spinning up a core when
> a request comes in for it.
>
> -Yonik
> http://lucidworks.com
>
> On Tue, Aug 13, 2013 at 4:26 PM, Erick Erickson <er...@gmail.com>
> wrote:
> > There was a question on the user's list today about making lazily-loaded
> > (aka transient) cores work with SolrCloud where I basically punted and
> said
> > "not designed with that in mind". I've kind of avoided thinking about
> this
> > as the use-case; the transient code wasn't written with SolrCloud in
> mind.
> >
> > But what is the general reaction to that pairing? Mostly I'm looking for
> > feedback at the level of "no way that could work without invasive
> changes to
> > SolrCloud, don't even go there" or "sure, just allow ZK to get a list of
> all
> > cores and it'll be fine, the user is responsible for the quirks though".
> > Some questions that come to my mind:
> >
> >> Is a core that's not loaded be considered "live" by ZK? Would simply
> >> returning a list of all cores (both loaded and not loaded) be
> sufficient for
> >> ZK? (this list is already available so the admin UI can list all cores).
> >
> >> Does SolrCloud distributed update processing go through (or could be
> made
> >> to go through) the path that autoloads a core?
> >
> >> Ditto for querying. I suspect the answer to both is that it'll "just
> >> happen".
> >
> >> Would the idea of waiting for all the cores to load on all the nodes for
> >> an update be totally unacceptable? We already have the distributed
> deadlock
> >> potential, this seems to make that more likely by lengthening out the
> time
> >> the semaphore in question is held.
> >
> >> Would re-synching/leader election be an absolute nightmare? I can
> imagine
> >> that if all the cores for a particular shard weren't loaded at startup,
> >> there'd be a terrible time waiting for leader election for instance.
> >
> >> Stuff I haven't thought of
> >
> > Mostly I'm trying to get a "sense of the community" here about whether
> > supporting transient cores in SolrCloud mode would be something that
> would
> > be easy/do-able/really_hard/totally_unacceptable.
> >
> > Thanks,
> > Erick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Re: lazily-loaded cores and SolrCloud

[Yonik Seeley <yo...@lucidworks.com>]

At a high level, I think the idea is fine (and I've seen a number of
people that wanted it).
The question is more around one of implementation... would it make a
mess of things or not.
The answer to that I think is probably mostly related to issues around
how zookeeper is currently handled.
I don't see any issues with other things like spinning up a core when
a request comes in for it.

-Yonik
http://lucidworks.com

On Tue, Aug 13, 2013 at 4:26 PM, Erick Erickson <er...@gmail.com> wrote:
> There was a question on the user's list today about making lazily-loaded
> (aka transient) cores work with SolrCloud where I basically punted and said
> "not designed with that in mind". I've kind of avoided thinking about this
> as the use-case; the transient code wasn't written with SolrCloud in mind.
>
> But what is the general reaction to that pairing? Mostly I'm looking for
> feedback at the level of "no way that could work without invasive changes to
> SolrCloud, don't even go there" or "sure, just allow ZK to get a list of all
> cores and it'll be fine, the user is responsible for the quirks though".
> Some questions that come to my mind:
>
>> Is a core that's not loaded be considered "live" by ZK? Would simply
>> returning a list of all cores (both loaded and not loaded) be sufficient for
>> ZK? (this list is already available so the admin UI can list all cores).
>
>> Does SolrCloud distributed update processing go through (or could be made
>> to go through) the path that autoloads a core?
>
>> Ditto for querying. I suspect the answer to both is that it'll "just
>> happen".
>
>> Would the idea of waiting for all the cores to load on all the nodes for
>> an update be totally unacceptable? We already have the distributed deadlock
>> potential, this seems to make that more likely by lengthening out the time
>> the semaphore in question is held.
>
>> Would re-synching/leader election be an absolute nightmare? I can imagine
>> that if all the cores for a particular shard weren't loaded at startup,
>> there'd be a terrible time waiting for leader election for instance.
>
>> Stuff I haven't thought of
>
> Mostly I'm trying to get a "sense of the community" here about whether
> supporting transient cores in SolrCloud mode would be something that would
> be easy/do-able/really_hard/totally_unacceptable.
>
> Thanks,
> Erick

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org