You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by John Blythe <jo...@curvolabs.com> on 2016/12/16 13:26:01 UTC

cores vs. instances vs. zookeeper vs. cloud vs ?

good morning everyone.

i've got a crowing number of cores that various parts of our application
are relying upon. i'm having difficulty figuring out the best way to
continue expanding for both sake of scale and convenience.

i need two extra versions of each core due to our demo instance and our
development instance. when we had just one or two cores it wasn't the worst
thing to have cores like X, demo-X, and dev-X. that has quickly become
unnecessarily cumbersome.

i've considered moving each instance to its own solr instance, perhaps just
throwing it on a different port. for example, production could be
x.x.x.x:8983, dev x.x.x.x:8993, and demo x.x.x.x:8938.

i'm pretty helpless at this point with zookeeper and/or solrcloud. given
the above info, i'd love to hear some quick overview ideas as to the best
approach that i can then begin to explore online.

thanks for any pointers!

Re: cores vs. instances vs. zookeeper vs. cloud vs ?

Posted by John Blythe <jo...@curvolabs.com>.
this is awesome. i'm jazzed to try it out. will do some introductory
examples and what not to familiarize myself and get the lightbulb to
(hopefully) go off. one last question: given all the info and questions
above, does it seem like ZK is overkill at this point and i should focus my
efforts on solrcloud? seems like it to me, but hope to confirm that before
investing time in the wrong direction.

thanks!

-- 
*John Blythe*
Product Manager & Lead Developer

251.605.3071 | john@curvolabs.com
www.curvolabs.com

58 Adams Ave
Evansville, IN 47713

On Fri, Dec 16, 2016 at 1:59 PM, Erick Erickson <er...@gmail.com>
wrote:

> bq: if i start using SolrCloud i could have my current multi-core setup
>    (e.g. "transactions", "opportunities", etc.) exist within the
> appropriate
>    collection.
>
> I'd guess that cores == collections but....
> Are you reaching across from one core to another to
> satisfy your use-case? I.e. using "cross core joins"
> or anything similar? You have to do some careful
> placement of collections and co-locate the replicas
> for each collection. This is quite do-able via the collecitons
> API.
>
> bq: this seems to be the same with ZK, too?
>
> Kind of ignore ZK. From what you're describing you won't
> have 100s of nodes. Therefore ZK will just keep track of it
> all for you. One common misconception is that ZK is involved
> in indexing and querying. It's not, kind of. Once each Solr
> instance gets the current state of the network, the Solr
> instances don't need to reference ZK to update or query.
> It's only when nodes change state (are shut down and the
> like) that ZK gets involved in letting the Solr nodes know about
> the state change.
>
> bq: is the separate indexing
>    happening out of the box w Cloud or something it's even capable of?
>
> Totally. The unit of address is a "collection". So let's say you have
> a collection named transactions_dev. You send updates to
> http://any_solr_server:port/solr/transactions_dev/update.
> Queries similarly as
> http://any_solr_server:port/solr/transactions_dev/query
>
> Don't try to put cores in here, just think about collections. The
> actual _core_ is something like transactions_dev_shard1_replica1
> but you'll almost never reference it directly unless you're debugging or
> something.
>
> So what I'd recommend is just go through the getting started example
> and at some point it'll all suddenly start to make sense ;).
>
> The take-away is that much of what you're doing in terms of keeping
> track of cores and all that just goes away.
>
> Best,
> Erick
>
> On Fri, Dec 16, 2016 at 9:30 AM, John Blythe <jo...@curvolabs.com> wrote:
> > thanks, erick. this is helpful. a few questions for clarity's sake, but
> > first: nope, not using SolrCloud as of yet.
> >
> >    - if i start using SolrCloud i could have my current multi-core setup
> >    (e.g. "transactions", "opportunities", etc.) exist within the
> appropriate
> >    collection. so instead of dev-transactions i'd have a 'dev'
> collection that
> >    has a 'transactions' core inside of it?
> >    - this seems to be the same with ZK, too?
> >    - i'm totally fine w separate/diff indexing. the demo collection, for
> >    instance, *has* to be separate from production bc the data has been
> >    stitched together from various customers' accounts on prod and
> blinded so
> >    that we have avoid privacy issues and can have all the various goodies
> >    under one demo account rather than separate ones. is the separate
> indexing
> >    happening out of the box w Cloud or something it's even capable of?
> >
> > thanks again, erick!
> >
> > --
> > *John Blythe*
> > Product Manager & Lead Developer
> >
> > 251.605.3071 | john@curvolabs.com
> > www.curvolabs.com
> >
> > 58 Adams Ave
> > Evansville, IN 47713
> >
> > On Fri, Dec 16, 2016 at 11:38 AM, Erick Erickson <
> erickerickson@gmail.com>
> > wrote:
> >
> >> It's not quite clear to me whether you're using SolrCloud now or not, my
> >> guess is not. My guess here is that you _should_ move to SolrCloud and
> >> collections. Then, instead of thinking about "cores", you just think
> about
> >> collections. Where the replicas live then isn't something you have to
> >> manage
> >> in that case.
> >>
> >> There's a bit of a learning curve for Zookeeper, and a mental shift you
> >> have to make to not worry about cores at all, just trust Solr. That
> said,
> >> if you _want_ to explicitly manage where each and every core for each
> >> and every collection lives, that's easy with the collections API. Once
> you
> >> do make that shift, going back is painful ;)
> >>
> >> So the scenario is that you have three collections, prod, dev, demo.
> They
> >> all
> >> happen to use the same configset (which you keep in ZK). You have one
> >> zookeeper ensemble that the three collections reference. They can even
> >> all share the same machine if that machine has sufficient capacity.
> >>
> >> The deal here is that these are really completely independent; you'll
> have
> >> to index your content to each separately.
> >>
> >> But then your URL becomes x.x.x.x:8983/solr/prod, x.x.x.x:8983/solr/dev
> >> and the like.
> >>
> >> FWIW,
> >> Erick
> >>
> >> On Fri, Dec 16, 2016 at 5:26 AM, John Blythe <jo...@curvolabs.com>
> wrote:
> >> > good morning everyone.
> >> >
> >> > i've got a crowing number of cores that various parts of our
> application
> >> > are relying upon. i'm having difficulty figuring out the best way to
> >> > continue expanding for both sake of scale and convenience.
> >> >
> >> > i need two extra versions of each core due to our demo instance and
> our
> >> > development instance. when we had just one or two cores it wasn't the
> >> worst
> >> > thing to have cores like X, demo-X, and dev-X. that has quickly become
> >> > unnecessarily cumbersome.
> >> >
> >> > i've considered moving each instance to its own solr instance, perhaps
> >> just
> >> > throwing it on a different port. for example, production could be
> >> > x.x.x.x:8983, dev x.x.x.x:8993, and demo x.x.x.x:8938.
> >> >
> >> > i'm pretty helpless at this point with zookeeper and/or solrcloud.
> given
> >> > the above info, i'd love to hear some quick overview ideas as to the
> best
> >> > approach that i can then begin to explore online.
> >> >
> >> > thanks for any pointers!
> >>
>

Re: cores vs. instances vs. zookeeper vs. cloud vs ?

Posted by Erick Erickson <er...@gmail.com>.
bq: if i start using SolrCloud i could have my current multi-core setup
   (e.g. "transactions", "opportunities", etc.) exist within the appropriate
   collection.

I'd guess that cores == collections but....
Are you reaching across from one core to another to
satisfy your use-case? I.e. using "cross core joins"
or anything similar? You have to do some careful
placement of collections and co-locate the replicas
for each collection. This is quite do-able via the collecitons
API.

bq: this seems to be the same with ZK, too?

Kind of ignore ZK. From what you're describing you won't
have 100s of nodes. Therefore ZK will just keep track of it
all for you. One common misconception is that ZK is involved
in indexing and querying. It's not, kind of. Once each Solr
instance gets the current state of the network, the Solr
instances don't need to reference ZK to update or query.
It's only when nodes change state (are shut down and the
like) that ZK gets involved in letting the Solr nodes know about
the state change.

bq: is the separate indexing
   happening out of the box w Cloud or something it's even capable of?

Totally. The unit of address is a "collection". So let's say you have
a collection named transactions_dev. You send updates to
http://any_solr_server:port/solr/transactions_dev/update.
Queries similarly as
http://any_solr_server:port/solr/transactions_dev/query

Don't try to put cores in here, just think about collections. The
actual _core_ is something like transactions_dev_shard1_replica1
but you'll almost never reference it directly unless you're debugging or
something.

So what I'd recommend is just go through the getting started example
and at some point it'll all suddenly start to make sense ;).

The take-away is that much of what you're doing in terms of keeping
track of cores and all that just goes away.

Best,
Erick

On Fri, Dec 16, 2016 at 9:30 AM, John Blythe <jo...@curvolabs.com> wrote:
> thanks, erick. this is helpful. a few questions for clarity's sake, but
> first: nope, not using SolrCloud as of yet.
>
>    - if i start using SolrCloud i could have my current multi-core setup
>    (e.g. "transactions", "opportunities", etc.) exist within the appropriate
>    collection. so instead of dev-transactions i'd have a 'dev' collection that
>    has a 'transactions' core inside of it?
>    - this seems to be the same with ZK, too?
>    - i'm totally fine w separate/diff indexing. the demo collection, for
>    instance, *has* to be separate from production bc the data has been
>    stitched together from various customers' accounts on prod and blinded so
>    that we have avoid privacy issues and can have all the various goodies
>    under one demo account rather than separate ones. is the separate indexing
>    happening out of the box w Cloud or something it's even capable of?
>
> thanks again, erick!
>
> --
> *John Blythe*
> Product Manager & Lead Developer
>
> 251.605.3071 | john@curvolabs.com
> www.curvolabs.com
>
> 58 Adams Ave
> Evansville, IN 47713
>
> On Fri, Dec 16, 2016 at 11:38 AM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> It's not quite clear to me whether you're using SolrCloud now or not, my
>> guess is not. My guess here is that you _should_ move to SolrCloud and
>> collections. Then, instead of thinking about "cores", you just think about
>> collections. Where the replicas live then isn't something you have to
>> manage
>> in that case.
>>
>> There's a bit of a learning curve for Zookeeper, and a mental shift you
>> have to make to not worry about cores at all, just trust Solr. That said,
>> if you _want_ to explicitly manage where each and every core for each
>> and every collection lives, that's easy with the collections API. Once you
>> do make that shift, going back is painful ;)
>>
>> So the scenario is that you have three collections, prod, dev, demo. They
>> all
>> happen to use the same configset (which you keep in ZK). You have one
>> zookeeper ensemble that the three collections reference. They can even
>> all share the same machine if that machine has sufficient capacity.
>>
>> The deal here is that these are really completely independent; you'll have
>> to index your content to each separately.
>>
>> But then your URL becomes x.x.x.x:8983/solr/prod, x.x.x.x:8983/solr/dev
>> and the like.
>>
>> FWIW,
>> Erick
>>
>> On Fri, Dec 16, 2016 at 5:26 AM, John Blythe <jo...@curvolabs.com> wrote:
>> > good morning everyone.
>> >
>> > i've got a crowing number of cores that various parts of our application
>> > are relying upon. i'm having difficulty figuring out the best way to
>> > continue expanding for both sake of scale and convenience.
>> >
>> > i need two extra versions of each core due to our demo instance and our
>> > development instance. when we had just one or two cores it wasn't the
>> worst
>> > thing to have cores like X, demo-X, and dev-X. that has quickly become
>> > unnecessarily cumbersome.
>> >
>> > i've considered moving each instance to its own solr instance, perhaps
>> just
>> > throwing it on a different port. for example, production could be
>> > x.x.x.x:8983, dev x.x.x.x:8993, and demo x.x.x.x:8938.
>> >
>> > i'm pretty helpless at this point with zookeeper and/or solrcloud. given
>> > the above info, i'd love to hear some quick overview ideas as to the best
>> > approach that i can then begin to explore online.
>> >
>> > thanks for any pointers!
>>

Re: cores vs. instances vs. zookeeper vs. cloud vs ?

Posted by John Blythe <jo...@curvolabs.com>.
ahhhh, there's a lightbulb. your last paragraph cleared up confusion i was
carrying through the responses (and, incidentally, was likely the reason
for confusion on the ZK question you couldn't make sense of). i was
thinking zookeeper was a separate means of handling things from solrcloud,
two entirely different approaches to scaling out. very much helpful to see
how off balance i was on that assumption!

thanks shawn-

-- 
*John Blythe*
Product Manager & Lead Developer

251.605.3071 | john@curvolabs.com
www.curvolabs.com

58 Adams Ave
Evansville, IN 47713

On Fri, Dec 16, 2016 at 3:31 PM, Shawn Heisey <ap...@elyograg.org> wrote:

> On 12/16/2016 10:30 AM, John Blythe wrote:
> > thanks, erick. this is helpful. a few questions for clarity's sake, but
> > first: nope, not using SolrCloud as of yet.
> >
> >    - if i start using SolrCloud i could have my current multi-core setup
> >    (e.g. "transactions", "opportunities", etc.) exist within the
> appropriate
> >    collection. so instead of dev-transactions i'd have a 'dev'
> collection that
> >    has a 'transactions' core inside of it?
>
> No.  You would not be thinking in terms of cores at all.  When your
> programs talk to SolrCloud, they will only care about collections.
>
> Some terminology clarification: Collections are made up of one or more
> shards.  Shards are made up of one or more replicas.  Each shard replica
> is a core.  One replica for each shard is elected as leader.  If there's
> only one replica, then there's no redundancy, and that replica becomes
> leader.
>
> For the example you gave, you would have dev-transactions and
> prod-transactions collections.  Each of these collections might have
> shard replicas (cores) on completely different machines in the cloud ...
> or they might be on the same machines.  During normal operation, you
> would never access a core directly.  You'd probably only ever do that if
> something went very wrong and you needed to take very unusual steps to
> fix it or figure out what went wrong.
>
> >    - this seems to be the same with ZK, too?
>
> No idea what you're asking here.  Perhaps it should be obvious, but I
> can't figure it out.
>
> >    - i'm totally fine w separate/diff indexing. the demo collection, for
> >    instance, *has* to be separate from production bc the data has been
> >    stitched together from various customers' accounts on prod and
> blinded so
> >    that we have avoid privacy issues and can have all the various goodies
> >    under one demo account rather than separate ones. is the separate
> indexing
> >    happening out of the box w Cloud or something it's even capable of?
>
> Again, I don't really know what you're asking with "separate indexing".
> Different collections are separate from each other, just like cores in
> standalone mode.  Each collection is linked to a configuration in
> Zookeeper, which all of its shard replicas (cores) will use.  You could
> have all your collections pointing to the same config.  Some (or all) of
> them could point to completely different configs, too.
>
> Addressing a later question:  You don't have SolrCloud if you don't have
> zookeeper.  ZK is a requirement.  You don't really interact directly
> with zookeeper when you're using SolrCloud.  It's an administrative
> detail in the *setup* of SolrCloud.
>
> Thanks,
> Shawn
>
>

Re: cores vs. instances vs. zookeeper vs. cloud vs ?

Posted by Shawn Heisey <ap...@elyograg.org>.
On 12/16/2016 10:30 AM, John Blythe wrote:
> thanks, erick. this is helpful. a few questions for clarity's sake, but
> first: nope, not using SolrCloud as of yet.
>
>    - if i start using SolrCloud i could have my current multi-core setup
>    (e.g. "transactions", "opportunities", etc.) exist within the appropriate
>    collection. so instead of dev-transactions i'd have a 'dev' collection that
>    has a 'transactions' core inside of it?

No.  You would not be thinking in terms of cores at all.  When your
programs talk to SolrCloud, they will only care about collections.

Some terminology clarification: Collections are made up of one or more
shards.  Shards are made up of one or more replicas.  Each shard replica
is a core.  One replica for each shard is elected as leader.  If there's
only one replica, then there's no redundancy, and that replica becomes
leader.

For the example you gave, you would have dev-transactions and
prod-transactions collections.  Each of these collections might have
shard replicas (cores) on completely different machines in the cloud ...
or they might be on the same machines.  During normal operation, you
would never access a core directly.  You'd probably only ever do that if
something went very wrong and you needed to take very unusual steps to
fix it or figure out what went wrong.

>    - this seems to be the same with ZK, too?

No idea what you're asking here.  Perhaps it should be obvious, but I
can't figure it out.

>    - i'm totally fine w separate/diff indexing. the demo collection, for
>    instance, *has* to be separate from production bc the data has been
>    stitched together from various customers' accounts on prod and blinded so
>    that we have avoid privacy issues and can have all the various goodies
>    under one demo account rather than separate ones. is the separate indexing
>    happening out of the box w Cloud or something it's even capable of?

Again, I don't really know what you're asking with "separate indexing". 
Different collections are separate from each other, just like cores in
standalone mode.  Each collection is linked to a configuration in
Zookeeper, which all of its shard replicas (cores) will use.  You could
have all your collections pointing to the same config.  Some (or all) of
them could point to completely different configs, too.

Addressing a later question:  You don't have SolrCloud if you don't have
zookeeper.  ZK is a requirement.  You don't really interact directly
with zookeeper when you're using SolrCloud.  It's an administrative
detail in the *setup* of SolrCloud.

Thanks,
Shawn


Re: cores vs. instances vs. zookeeper vs. cloud vs ?

Posted by John Blythe <jo...@curvolabs.com>.
thanks, erick. this is helpful. a few questions for clarity's sake, but
first: nope, not using SolrCloud as of yet.

   - if i start using SolrCloud i could have my current multi-core setup
   (e.g. "transactions", "opportunities", etc.) exist within the appropriate
   collection. so instead of dev-transactions i'd have a 'dev' collection that
   has a 'transactions' core inside of it?
   - this seems to be the same with ZK, too?
   - i'm totally fine w separate/diff indexing. the demo collection, for
   instance, *has* to be separate from production bc the data has been
   stitched together from various customers' accounts on prod and blinded so
   that we have avoid privacy issues and can have all the various goodies
   under one demo account rather than separate ones. is the separate indexing
   happening out of the box w Cloud or something it's even capable of?

thanks again, erick!

-- 
*John Blythe*
Product Manager & Lead Developer

251.605.3071 | john@curvolabs.com
www.curvolabs.com

58 Adams Ave
Evansville, IN 47713

On Fri, Dec 16, 2016 at 11:38 AM, Erick Erickson <er...@gmail.com>
wrote:

> It's not quite clear to me whether you're using SolrCloud now or not, my
> guess is not. My guess here is that you _should_ move to SolrCloud and
> collections. Then, instead of thinking about "cores", you just think about
> collections. Where the replicas live then isn't something you have to
> manage
> in that case.
>
> There's a bit of a learning curve for Zookeeper, and a mental shift you
> have to make to not worry about cores at all, just trust Solr. That said,
> if you _want_ to explicitly manage where each and every core for each
> and every collection lives, that's easy with the collections API. Once you
> do make that shift, going back is painful ;)
>
> So the scenario is that you have three collections, prod, dev, demo. They
> all
> happen to use the same configset (which you keep in ZK). You have one
> zookeeper ensemble that the three collections reference. They can even
> all share the same machine if that machine has sufficient capacity.
>
> The deal here is that these are really completely independent; you'll have
> to index your content to each separately.
>
> But then your URL becomes x.x.x.x:8983/solr/prod, x.x.x.x:8983/solr/dev
> and the like.
>
> FWIW,
> Erick
>
> On Fri, Dec 16, 2016 at 5:26 AM, John Blythe <jo...@curvolabs.com> wrote:
> > good morning everyone.
> >
> > i've got a crowing number of cores that various parts of our application
> > are relying upon. i'm having difficulty figuring out the best way to
> > continue expanding for both sake of scale and convenience.
> >
> > i need two extra versions of each core due to our demo instance and our
> > development instance. when we had just one or two cores it wasn't the
> worst
> > thing to have cores like X, demo-X, and dev-X. that has quickly become
> > unnecessarily cumbersome.
> >
> > i've considered moving each instance to its own solr instance, perhaps
> just
> > throwing it on a different port. for example, production could be
> > x.x.x.x:8983, dev x.x.x.x:8993, and demo x.x.x.x:8938.
> >
> > i'm pretty helpless at this point with zookeeper and/or solrcloud. given
> > the above info, i'd love to hear some quick overview ideas as to the best
> > approach that i can then begin to explore online.
> >
> > thanks for any pointers!
>

Re: cores vs. instances vs. zookeeper vs. cloud vs ?

Posted by Erick Erickson <er...@gmail.com>.
It's not quite clear to me whether you're using SolrCloud now or not, my
guess is not. My guess here is that you _should_ move to SolrCloud and
collections. Then, instead of thinking about "cores", you just think about
collections. Where the replicas live then isn't something you have to manage
in that case.

There's a bit of a learning curve for Zookeeper, and a mental shift you
have to make to not worry about cores at all, just trust Solr. That said,
if you _want_ to explicitly manage where each and every core for each
and every collection lives, that's easy with the collections API. Once you
do make that shift, going back is painful ;)

So the scenario is that you have three collections, prod, dev, demo. They all
happen to use the same configset (which you keep in ZK). You have one
zookeeper ensemble that the three collections reference. They can even
all share the same machine if that machine has sufficient capacity.

The deal here is that these are really completely independent; you'll have
to index your content to each separately.

But then your URL becomes x.x.x.x:8983/solr/prod, x.x.x.x:8983/solr/dev
and the like.

FWIW,
Erick

On Fri, Dec 16, 2016 at 5:26 AM, John Blythe <jo...@curvolabs.com> wrote:
> good morning everyone.
>
> i've got a crowing number of cores that various parts of our application
> are relying upon. i'm having difficulty figuring out the best way to
> continue expanding for both sake of scale and convenience.
>
> i need two extra versions of each core due to our demo instance and our
> development instance. when we had just one or two cores it wasn't the worst
> thing to have cores like X, demo-X, and dev-X. that has quickly become
> unnecessarily cumbersome.
>
> i've considered moving each instance to its own solr instance, perhaps just
> throwing it on a different port. for example, production could be
> x.x.x.x:8983, dev x.x.x.x:8993, and demo x.x.x.x:8938.
>
> i'm pretty helpless at this point with zookeeper and/or solrcloud. given
> the above info, i'd love to hear some quick overview ideas as to the best
> approach that i can then begin to explore online.
>
> thanks for any pointers!