You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Per Steffensen <st...@designware.dk> on 2012/11/26 10:57:21 UTC

Multiple shards for one collection on the same Solr server

Hi

Before upgrading to Solr 4.0.0 we used to handle our collection creation 
ourselves, by creating each shards through the low-level CoreAdmin API. 
We used to create multiple shards under the same collection on each Solr 
server. Performance tests has shown that this is a good idea, and it is 
also a good idea for easy elasticity later on - it is much easier to 
move an entire existing shards from one Solr server to another one that 
just joined the cluter than it is to split an exsiting shard among the 
Solr that used to run it and the new Solr.

Now we are trying to migrate to the Solr Collection API for creation of 
collections, but it seems like it will not accept multiple shards under 
the same collection running on the same Solr server. E.g. if we have 4 
Solr servers and ask to have a collection created with 8 shards, all 8 
shards will be "created" but only 4 of them will acutally run - one on 
each Solr server.

Is the a good reason why Solr does not allow multiple shards under the 
same collection to run on the same Solr server, or is it just made this 
way "by coincidence"? In general I seek info on the matter - is it 
planed for later? etc.

Thanks!

Regards, Per Steffensen

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Multiple shards for one collection on the same Solr server

Posted by Otis Gospodnetic <ot...@gmail.com>.
Per - keep contributing.  Your contributions so far have not gone unnoticed
- you've been spotted.  At least from here in New York. :)

Otis
--
SOLR Performance Monitoring - http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-analytics/index.html




On Mon, Nov 26, 2012 at 2:43 PM, Per Steffensen <st...@designware.dk> wrote:

> **
> Ok thanks. Guess you are not ready to nominate me for now. But who are the
> nominators? Are you a nominator, Mark? And how to get nominated - do I have
> to ask (as I did today) or do I just have to wait until a nominator gets
> the idea about me as a committer by himself?
>
>
> Regards, Per Steffensen
>
> Mark Miller skrev:
>
> On Nov 26, 2012, at 11:51 AM, Per Steffensen <st...@designware.dk> <st...@designware.dk> wrote:
>
>
>
>  I am a very experinced programmer with a long record as architect, designer, mentor etc. Is there a chance that I could become committer? You will end up happy - and so will I! :-)
>
>
>  You have to build up enough merit in the project to get someone to nominate you. Getting someone to nominate you can come about in a few different ways, but generally it requires seeing a good number of JIRA issues through, being around long enough to learn and integrate with the current community vibe, and sometimes other things like helping out on the mailing list, helping others get JIRA issues in, etc.
>
> You have to build the merit in the community - merit doesn't transfer from outside of the project.
>
> You can imagine why we are careful about this - we want to make sure the community grows in a healthy manner and that the others we bring into the community are a nice fit with the rest of us. We want to make sure some of the Apache way has seeped into your mind and certainly that a lot of the current community vibe has as well. IMO, the best committers tend to act like committers before they are voted in. Once you earn the trust of another committer or two, and contribute in digestible pieces, you can generally build a relationship that gets your patches committed quite quickly. Once a committer comes to trust your code, he tends to have an easier time reviewing. Then he starts committing more of your stuff. Then he thinks, man, this would just be easier if this guy was a committer.
>
> In any event, simply continuing to contribute and getting issues into Solr over a decent length of time is generally enough to be nominated at some point. Though it really all depends on the nominators in the end. And then of course you must be voted in by the PMC.
>
> - Mark
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
>
>

Re: Multiple shards for one collection on the same Solr server

Posted by Mark Miller <ma...@gmail.com>.
The nominators are generally the committers on the lucene and solr projects. There is a list of those on the website under "Who We Are". Yes, I am one. The PMC then votes - there is also a list of who is on the PMC on the website.

The history is, generally you just wait to be nominated. Everyone has their own criteria for nominating and voting. Because most were brought in using similar criteria, people tend to use the same criteria when they then start nominating others. There is something of a culture we are attempting to guide. 

- Mark

On Nov 26, 2012, at 2:43 PM, Per Steffensen <st...@designware.dk> wrote:

> Ok thanks. Guess you are not ready to nominate me for now. But who are the nominators? Are you a nominator, Mark? And how to get nominated - do I have to ask (as I did today) or do I just have to wait until a nominator gets the idea about me as a committer by himself?
> 
> Regards, Per Steffensen
> 
> Mark Miller skrev:
>> On Nov 26, 2012, at 11:51 AM, Per Steffensen <st...@designware.dk>
>>  wrote:
>> 
>>   
>> 
>>> I am a very experinced programmer with a long record as architect, designer, mentor etc. Is there a chance that I could become committer? You will end up happy - and so will I! :-)
>>>     
>>> 
>> 
>> You have to build up enough merit in the project to get someone to nominate you. Getting someone to nominate you can come about in a few different ways, but generally it requires seeing a good number of JIRA issues through, being around long enough to learn and integrate with the current community vibe, and sometimes other things like helping out on the mailing list, helping others get JIRA issues in, etc.
>> 
>> You have to build the merit in the community - merit doesn't transfer from outside of the project.
>> 
>> You can imagine why we are careful about this - we want to make sure the community grows in a healthy manner and that the others we bring into the community are a nice fit with the rest of us. We want to make sure some of the Apache way has seeped into your mind and certainly that a lot of the current community vibe has as well. IMO, the best committers tend to act like committers before they are voted in. Once you earn the trust of another committer or two, and contribute in digestible pieces, you can generally build a relationship that gets your patches committed quite quickly. Once a committer comes to trust your code, he tends to have an easier time reviewing. Then he starts committing more of your stuff. Then he thinks, man, this would just be easier if this guy was a committer.
>> 
>> In any event, simply continuing to contribute and getting issues into Solr over a decent length of time is generally enough to be nominated at some point. Though it really all depends on the nominators in the end. And then of course you must be voted in by the PMC.
>> 
>> - Mark
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: 
>> dev-unsubscribe@lucene.apache.org
>> 
>> For additional commands, e-mail: 
>> dev-help@lucene.apache.org
>> 
>> 
>> 
>>   
>> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Multiple shards for one collection on the same Solr server

Posted by Per Steffensen <st...@designware.dk>.
Ok thanks. Guess you are not ready to nominate me for now. But who are 
the nominators? Are you a nominator, Mark? And how to get nominated - do 
I have to ask (as I did today) or do I just have to wait until a 
nominator gets the idea about me as a committer by himself?

Regards, Per Steffensen

Mark Miller skrev:
> On Nov 26, 2012, at 11:51 AM, Per Steffensen <st...@designware.dk> wrote:
>
>   
>> I am a very experinced programmer with a long record as architect, designer, mentor etc. Is there a chance that I could become committer? You will end up happy - and so will I! :-)
>>     
>
> You have to build up enough merit in the project to get someone to nominate you. Getting someone to nominate you can come about in a few different ways, but generally it requires seeing a good number of JIRA issues through, being around long enough to learn and integrate with the current community vibe, and sometimes other things like helping out on the mailing list, helping others get JIRA issues in, etc.
>
> You have to build the merit in the community - merit doesn't transfer from outside of the project.
>
> You can imagine why we are careful about this - we want to make sure the community grows in a healthy manner and that the others we bring into the community are a nice fit with the rest of us. We want to make sure some of the Apache way has seeped into your mind and certainly that a lot of the current community vibe has as well. IMO, the best committers tend to act like committers before they are voted in. Once you earn the trust of another committer or two, and contribute in digestible pieces, you can generally build a relationship that gets your patches committed quite quickly. Once a committer comes to trust your code, he tends to have an easier time reviewing. Then he starts committing more of your stuff. Then he thinks, man, this would just be easier if this guy was a committer.
>
> In any event, simply continuing to contribute and getting issues into Solr over a decent length of time is generally enough to be nominated at some point. Though it really all depends on the nominators in the end. And then of course you must be voted in by the PMC.
>
> - Mark
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
>   


Re: Multiple shards for one collection on the same Solr server

Posted by Mark Miller <ma...@gmail.com>.
On Nov 26, 2012, at 11:51 AM, Per Steffensen <st...@designware.dk> wrote:

> I am a very experinced programmer with a long record as architect, designer, mentor etc. Is there a chance that I could become committer? You will end up happy - and so will I! :-)

You have to build up enough merit in the project to get someone to nominate you. Getting someone to nominate you can come about in a few different ways, but generally it requires seeing a good number of JIRA issues through, being around long enough to learn and integrate with the current community vibe, and sometimes other things like helping out on the mailing list, helping others get JIRA issues in, etc.

You have to build the merit in the community - merit doesn't transfer from outside of the project.

You can imagine why we are careful about this - we want to make sure the community grows in a healthy manner and that the others we bring into the community are a nice fit with the rest of us. We want to make sure some of the Apache way has seeped into your mind and certainly that a lot of the current community vibe has as well. IMO, the best committers tend to act like committers before they are voted in. Once you earn the trust of another committer or two, and contribute in digestible pieces, you can generally build a relationship that gets your patches committed quite quickly. Once a committer comes to trust your code, he tends to have an easier time reviewing. Then he starts committing more of your stuff. Then he thinks, man, this would just be easier if this guy was a committer.

In any event, simply continuing to contribute and getting issues into Solr over a decent length of time is generally enough to be nominated at some point. Though it really all depends on the nominators in the end. And then of course you must be voted in by the PMC.

- Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Multiple shards for one collection on the same Solr server

Posted by Per Steffensen <st...@designware.dk>.
Of course I understand everything you say. We are just using Solr very 
intensively in my current project and we want to do big changes and when 
we make them we test them very thoroughly - so we are pretty sure they 
work as they are supposed to, but of course that is hard to see on your 
side, because we do not make that many tests in the Solr code, because 
that would make our contribs even bigger. But we test them thoroughly 
through our own test-suite. I have worked with Solr for a long time now 
and there is a million things that I (we in my project in general) would 
like to do, we just have a hard time to really get some progress - and 
my POs face turns green whenever I say that a feature/fix of some issue 
will require a Solr modification, because he knows how hard it is for us 
to get contribs through, and he knowns that the amount of time we use 
merging every time we want to get the newest code from Apache will 
increase by the amount of code we have different from yours.

I am a very experinced programmer with a long record as architect, 
designer, mentor etc. Is there a chance that I could become committer? 
You will end up happy - and so will I! :-)

Regards, Per Steffensen

Mark Miller skrev:
> On Mon, Nov 26, 2012 at 9:55 AM, Per Steffensen <st...@designware.dk> wrote:
>   
>> Well I would like to jump in with respect to making support for running
>> several shards of the same collection on the same instance, it is just so
>> damn hard to get you to commit stuff :-)
>>     
>
> A lot of the problem is time limitations for me personally. Every day
> I go to sleep late wishing I could stay up just a little later :) I
> never get everything I want done and I have a back log list of things
> I'm interested in a mile long. So I sympathize, but I'm stretched thin
> personally.
>
> I do think though, if you look at CHANGES.txt, you will see that I do
> commit a lot of other peoples patches! They just tend to be targeted
> things that are relatively easy to digest. Also, things that are on my
> itch list.
>
> The main patches you have supplied so far are quite large! And touch
> areas that involve other committers. Large patches and multiple
> parties do not generally do well as we have mentioned in the past. It
> depends on the committers, their time, their agenda, their itch list,
> timing, etc.
>
> I wouldn't give up trying to contribute though. Many of the things
> tied up in your larger issues are things I'd like to get too. Once or
> twice I've already gone into your work to 'steal' a bit when
> addressing issues (once I got the time).
>
> I know you'd prefer we were looser about committing if tests just
> pass, but it comes down to what each committer is comfortable taking
> responsibility for - and I'm not willing to generally commit larger
> patches like that unless I have thoroughly vetted them - something
> that takes a lot of my time.
>
> I hope you continue contributing in any case!
>
>   


Re: Multiple shards for one collection on the same Solr server

Posted by Mark Miller <ma...@gmail.com>.
On Mon, Nov 26, 2012 at 9:55 AM, Per Steffensen <st...@designware.dk> wrote:
> Well I would like to jump in with respect to making support for running
> several shards of the same collection on the same instance, it is just so
> damn hard to get you to commit stuff :-)

A lot of the problem is time limitations for me personally. Every day
I go to sleep late wishing I could stay up just a little later :) I
never get everything I want done and I have a back log list of things
I'm interested in a mile long. So I sympathize, but I'm stretched thin
personally.

I do think though, if you look at CHANGES.txt, you will see that I do
commit a lot of other peoples patches! They just tend to be targeted
things that are relatively easy to digest. Also, things that are on my
itch list.

The main patches you have supplied so far are quite large! And touch
areas that involve other committers. Large patches and multiple
parties do not generally do well as we have mentioned in the past. It
depends on the committers, their time, their agenda, their itch list,
timing, etc.

I wouldn't give up trying to contribute though. Many of the things
tied up in your larger issues are things I'd like to get too. Once or
twice I've already gone into your work to 'steal' a bit when
addressing issues (once I got the time).

I know you'd prefer we were looser about committing if tests just
pass, but it comes down to what each committer is comfortable taking
responsibility for - and I'm not willing to generally commit larger
patches like that unless I have thoroughly vetted them - something
that takes a lot of my time.

I hope you continue contributing in any case!

-- 
- Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Multiple shards for one collection on the same Solr server

Posted by Per Steffensen <st...@designware.dk>.
Well I have created issue SOLR-4114 on the subject. Patch comming up.

Regards, Per Steffensen

Per Steffensen skrev:
> Mark Miller skrev:
>> The Collections API was fairly rushed - so that 4.0 had something easier than the CoreAdmin API.
>>   
> Yes I see. Our collection-creation code is more sophisticated than 
> yours. We probably would like to migrate to the Solr Collection API 
> now anyway - to be using it already when features are added later.
>> Due to that, it has a variety of limitations:
>>
>> 1. It only picks instances for a collection one way - randomly from the list of live instances. This means it's no good for multiple shards on the same instance. You should have enough instances to satisfy numShards X replicationFactor (although just being short on replicationFactor will currently just use what is there)
>>   
> Well I think it shuffles the list of live-nodes and the begin 
> assigning shard from one end. That is ok for us for now. But it will 
> not start over in the list of live-nodes when there are more shards 
> (shards * replica) than instances. This could easily be acheived, 
> without making a very fancy allocation algorithm
>> 2. It randomly chooses which instances to use rather than allowing manual specification or looking at existing cores.
>>   
> A manual spec would be nice to be able to control everything if you 
> really want to. But you probably also want to make different built-in 
> shard-allocation-strategies that can be used out-of-the-box. E.g. a 
> "AlwaysAssignNextShardToInstanceWithFewestShardsAlready"-strategy, but 
> there are also other concerns that might be more interesting for 
> people to have build into assignment algorithms - e.g. a rack-aware 
> algorithm that assign replica of the same slice to instances running 
> on different "racks".
>> 3. You cannot get responses of success or failure other than polling for the expected results later.
>>   
> Well we do that anyway, and will keep doing that in our own code for now.
>> Someone has a patch up for 3 that I hope to look at soon - others have contributed bug fixes that will be in 4.1. We still need to add the ability to control placement in other ways though.
>>
>> I would say there are def plans, but I don't personally know exactly when I'll find the time for it, if others don't jump in.
>>   
> Well I would like to jump in with respect to making support for 
> running several shards of the same collection on the same instance, it 
> is just so damn hard to get you to commit stuff :-) and we really dont 
> want to have too many differences in our Solr compared to Apache Solr 
> (and we have enough already - SOLR-3178 etc.). It seems like this 
> feature with several shards on same instance is the only missing 
> feature of the Collection API before we can "live with it".
>> - Mark
>>   
> Regards, Per Steffensen


Re: Multiple shards for one collection on the same Solr server

Posted by Per Steffensen <st...@designware.dk>.
Mark Miller skrev:
> The Collections API was fairly rushed - so that 4.0 had something easier than the CoreAdmin API.
>   
Yes I see. Our collection-creation code is more sophisticated than 
yours. We probably would like to migrate to the Solr Collection API now 
anyway - to be using it already when features are added later.
> Due to that, it has a variety of limitations:
>
> 1. It only picks instances for a collection one way - randomly from the list of live instances. This means it's no good for multiple shards on the same instance. You should have enough instances to satisfy numShards X replicationFactor (although just being short on replicationFactor will currently just use what is there)
>   
Well I think it shuffles the list of live-nodes and the begin assigning 
shard from one end. That is ok for us for now. But it will not start 
over in the list of live-nodes when there are more shards (shards * 
replica) than instances. This could easily be acheived, without making a 
very fancy allocation algorithm
> 2. It randomly chooses which instances to use rather than allowing manual specification or looking at existing cores.
>   
A manual spec would be nice to be able to control everything if you 
really want to. But you probably also want to make different built-in 
shard-allocation-strategies that can be used out-of-the-box. E.g. a 
"AlwaysAssignNextShardToInstanceWithFewestShardsAlready"-strategy, but 
there are also other concerns that might be more interesting for people 
to have build into assignment algorithms - e.g. a rack-aware algorithm 
that assign replica of the same slice to instances running on different 
"racks".
> 3. You cannot get responses of success or failure other than polling for the expected results later.
>   
Well we do that anyway, and will keep doing that in our own code for now.
>
> Someone has a patch up for 3 that I hope to look at soon - others have contributed bug fixes that will be in 4.1. We still need to add the ability to control placement in other ways though.
>
> I would say there are def plans, but I don't personally know exactly when I'll find the time for it, if others don't jump in.
>   
Well I would like to jump in with respect to making support for running 
several shards of the same collection on the same instance, it is just 
so damn hard to get you to commit stuff :-) and we really dont want to 
have too many differences in our Solr compared to Apache Solr (and we 
have enough already - SOLR-3178 etc.). It seems like this feature with 
several shards on same instance is the only missing feature of the 
Collection API before we can "live with it".
> - Mark
>   
Regards, Per Steffensen
> On Nov 26, 2012, at 4:57 AM, Per Steffensen <st...@designware.dk> wrote:
>
>   
>> Hi
>>
>> Before upgrading to Solr 4.0.0 we used to handle our collection creation ourselves, by creating each shards through the low-level CoreAdmin API. We used to create multiple shards under the same collection on each Solr server. Performance tests has shown that this is a good idea, and it is also a good idea for easy elasticity later on - it is much easier to move an entire existing shards from one Solr server to another one that just joined the cluter than it is to split an exsiting shard among the Solr that used to run it and the new Solr.
>>
>> Now we are trying to migrate to the Solr Collection API for creation of collections, but it seems like it will not accept multiple shards under the same collection running on the same Solr server. E.g. if we have 4 Solr servers and ask to have a collection created with 8 shards, all 8 shards will be "created" but only 4 of them will acutally run - one on each Solr server.
>>
>> Is the a good reason why Solr does not allow multiple shards under the same collection to run on the same Solr server, or is it just made this way "by coincidence"? In general I seek info on the matter - is it planed for later? etc.
>>
>> Thanks!
>>
>> Regards, Per Steffensen
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>     
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
>   


Re: Multiple shards for one collection on the same Solr server

Posted by Mark Miller <ma...@gmail.com>.
The Collections API was fairly rushed - so that 4.0 had something easier than the CoreAdmin API.

Due to that, it has a variety of limitations:

1. It only picks instances for a collection one way - randomly from the list of live instances. This means it's no good for multiple shards on the same instance. You should have enough instances to satisfy numShards X replicationFactor (although just being short on replicationFactor will currently just use what is there)

2. It randomly chooses which instances to use rather than allowing manual specification or looking at existing cores.

3. You cannot get responses of success or failure other than polling for the expected results later.


Someone has a patch up for 3 that I hope to look at soon - others have contributed bug fixes that will be in 4.1. We still need to add the ability to control placement in other ways though.

I would say there are def plans, but I don't personally know exactly when I'll find the time for it, if others don't jump in.

- Mark

On Nov 26, 2012, at 4:57 AM, Per Steffensen <st...@designware.dk> wrote:

> Hi
> 
> Before upgrading to Solr 4.0.0 we used to handle our collection creation ourselves, by creating each shards through the low-level CoreAdmin API. We used to create multiple shards under the same collection on each Solr server. Performance tests has shown that this is a good idea, and it is also a good idea for easy elasticity later on - it is much easier to move an entire existing shards from one Solr server to another one that just joined the cluter than it is to split an exsiting shard among the Solr that used to run it and the new Solr.
> 
> Now we are trying to migrate to the Solr Collection API for creation of collections, but it seems like it will not accept multiple shards under the same collection running on the same Solr server. E.g. if we have 4 Solr servers and ask to have a collection created with 8 shards, all 8 shards will be "created" but only 4 of them will acutally run - one on each Solr server.
> 
> Is the a good reason why Solr does not allow multiple shards under the same collection to run on the same Solr server, or is it just made this way "by coincidence"? In general I seek info on the matter - is it planed for later? etc.
> 
> Thanks!
> 
> Regards, Per Steffensen
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org