You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jamie Johnson <je...@gmail.com> on 2012/02/10 15:33:17 UTC
SolrCloud Replication Question
I know that the latest Solr Cloud doesn't use standard replication but
I have a question about how it appears to be working. I currently
have the following cluster state
{"collection1":{
"slice1":{
"JamiesMac.local:8501_solr_slice1_shard1":{
"shard_id":"slice1",
"state":"active",
"core":"slice1_shard1",
"collection":"collection1",
"node_name":"JamiesMac.local:8501_solr",
"base_url":"http://JamiesMac.local:8501/solr"},
"JamiesMac.local:8502_solr_slice1_shard2":{
"shard_id":"slice1",
"state":"active",
"core":"slice1_shard2",
"collection":"collection1",
"node_name":"JamiesMac.local:8502_solr",
"base_url":"http://JamiesMac.local:8502/solr"},
"jamiesmac:8501_solr_slice1_shard1":{
"shard_id":"slice1",
"state":"down",
"core":"slice1_shard1",
"collection":"collection1",
"node_name":"jamiesmac:8501_solr",
"base_url":"http://jamiesmac:8501/solr"},
"jamiesmac:8502_solr_slice1_shard2":{
"shard_id":"slice1",
"leader":"true",
"state":"active",
"core":"slice1_shard2",
"collection":"collection1",
"node_name":"jamiesmac:8502_solr",
"base_url":"http://jamiesmac:8502/solr"}},
"slice2":{
"JamiesMac.local:8501_solr_slice2_shard2":{
"shard_id":"slice2",
"state":"active",
"core":"slice2_shard2",
"collection":"collection1",
"node_name":"JamiesMac.local:8501_solr",
"base_url":"http://JamiesMac.local:8501/solr"},
"JamiesMac.local:8502_solr_slice2_shard1":{
"shard_id":"slice2",
"state":"active",
"core":"slice2_shard1",
"collection":"collection1",
"node_name":"JamiesMac.local:8502_solr",
"base_url":"http://JamiesMac.local:8502/solr"},
"jamiesmac:8501_solr_slice2_shard2":{
"shard_id":"slice2",
"state":"down",
"core":"slice2_shard2",
"collection":"collection1",
"node_name":"jamiesmac:8501_solr",
"base_url":"http://jamiesmac:8501/solr"},
"jamiesmac:8502_solr_slice2_shard1":{
"shard_id":"slice2",
"leader":"true",
"state":"active",
"core":"slice2_shard1",
"collection":"collection1",
"node_name":"jamiesmac:8502_solr",
"base_url":"http://jamiesmac:8502/solr"}}}}
I then added some docs to the following shards using SolrJ
http://localhost:8502/solr/slice2_shard1
http://localhost:8502/solr/slice1_shard2
I then bring back up the other cores and I don't see replication
happening. Looking at the stats for each core I see that on the 8501
instance (the instance that was off) the number of docs is 0, so I've
clearly set something up incorrectly. Any help on this would be
greatly appreciated.
Re: SolrCloud Replication Question
Posted by Mark Miller <ma...@gmail.com>.
Thanks.
If the given ZK snapshot was the end state, then two nodes are marked as
down. Generally that happens because replication failed - if you have not,
I'd check the logs for those two nodes.
- Mark
On Fri, Feb 10, 2012 at 7:35 PM, Jamie Johnson <je...@gmail.com> wrote:
> nothing seems that different. In regards to the states of each I'll
> try to verify tonight.
>
> This was using a version I pulled from SVN trunk yesterday morning
>
> On Fri, Feb 10, 2012 at 6:22 PM, Mark Miller <ma...@gmail.com>
> wrote:
> > Also, it will help if you can mention the exact version of solrcloud you
> are talking about in each issue - I know you have one from the old branch,
> and I assume a version off trunk you are playing with - so a heads up on
> which and if trunk, what rev or day will help in the case that I'm trying
> to dupe issues that have been addressed.
> >
> > - Mark
> >
> > On Feb 10, 2012, at 6:09 PM, Mark Miller wrote:
> >
> >> I'm trying, but so far I don't see anything. I'll have to try and mimic
> your setup closer it seems.
> >>
> >> I tried starting up 6 solr instances on different ports as 2 shards,
> each with a replication factor of 3.
> >>
> >> Then I indexed 20k documents to the cluster and verified doc counts.
> >>
> >> Then I shutdown all the replicas so that only one instance served each
> shard.
> >>
> >> Then I indexed 20k documents to the cluster.
> >>
> >> Then I started the downed nodes and verified that they where in a
> recovery state.
> >>
> >> After enough time went by I checked and verified document counts on
> each instance - they where as expected.
> >>
> >> I guess next I can try a similar experiment using multiple cores, but
> if you notice anything that stands out that is largely different in what
> you are doing, let me know.
> >>
> >> The cores that are behind, does it say they are down, recovering, or
> active in zookeeper?
> >>
> >> On Feb 10, 2012, at 4:48 PM, Jamie Johnson wrote:
> >>
> >>> Sorry for pinging this again, is more information needed on this? I
> >>> can provide more details but am not sure what to provide.
> >>>
> >>> On Fri, Feb 10, 2012 at 10:26 AM, Jamie Johnson <je...@gmail.com>
> wrote:
> >>>> Sorry, I shut down the full solr instance.
> >>>>
> >>>> On Fri, Feb 10, 2012 at 9:42 AM, Mark Miller <ma...@gmail.com>
> wrote:
> >>>>> Can you explain a little more how you doing this? How are you
> bringing the cores down and then back up? Shutting down a full solr
> instance, unloading the core?
> >>>>>
> >>>>> On Feb 10, 2012, at 9:33 AM, Jamie Johnson wrote:
> >>>>>
> >>>>>> I know that the latest Solr Cloud doesn't use standard replication
> but
> >>>>>> I have a question about how it appears to be working. I currently
> >>>>>> have the following cluster state
> >>>>>>
> >>>>>> {"collection1":{
> >>>>>> "slice1":{
> >>>>>> "JamiesMac.local:8501_solr_slice1_shard1":{
> >>>>>> "shard_id":"slice1",
> >>>>>> "state":"active",
> >>>>>> "core":"slice1_shard1",
> >>>>>> "collection":"collection1",
> >>>>>> "node_name":"JamiesMac.local:8501_solr",
> >>>>>> "base_url":"http://JamiesMac.local:8501/solr"},
> >>>>>> "JamiesMac.local:8502_solr_slice1_shard2":{
> >>>>>> "shard_id":"slice1",
> >>>>>> "state":"active",
> >>>>>> "core":"slice1_shard2",
> >>>>>> "collection":"collection1",
> >>>>>> "node_name":"JamiesMac.local:8502_solr",
> >>>>>> "base_url":"http://JamiesMac.local:8502/solr"},
> >>>>>> "jamiesmac:8501_solr_slice1_shard1":{
> >>>>>> "shard_id":"slice1",
> >>>>>> "state":"down",
> >>>>>> "core":"slice1_shard1",
> >>>>>> "collection":"collection1",
> >>>>>> "node_name":"jamiesmac:8501_solr",
> >>>>>> "base_url":"http://jamiesmac:8501/solr"},
> >>>>>> "jamiesmac:8502_solr_slice1_shard2":{
> >>>>>> "shard_id":"slice1",
> >>>>>> "leader":"true",
> >>>>>> "state":"active",
> >>>>>> "core":"slice1_shard2",
> >>>>>> "collection":"collection1",
> >>>>>> "node_name":"jamiesmac:8502_solr",
> >>>>>> "base_url":"http://jamiesmac:8502/solr"}},
> >>>>>> "slice2":{
> >>>>>> "JamiesMac.local:8501_solr_slice2_shard2":{
> >>>>>> "shard_id":"slice2",
> >>>>>> "state":"active",
> >>>>>> "core":"slice2_shard2",
> >>>>>> "collection":"collection1",
> >>>>>> "node_name":"JamiesMac.local:8501_solr",
> >>>>>> "base_url":"http://JamiesMac.local:8501/solr"},
> >>>>>> "JamiesMac.local:8502_solr_slice2_shard1":{
> >>>>>> "shard_id":"slice2",
> >>>>>> "state":"active",
> >>>>>> "core":"slice2_shard1",
> >>>>>> "collection":"collection1",
> >>>>>> "node_name":"JamiesMac.local:8502_solr",
> >>>>>> "base_url":"http://JamiesMac.local:8502/solr"},
> >>>>>> "jamiesmac:8501_solr_slice2_shard2":{
> >>>>>> "shard_id":"slice2",
> >>>>>> "state":"down",
> >>>>>> "core":"slice2_shard2",
> >>>>>> "collection":"collection1",
> >>>>>> "node_name":"jamiesmac:8501_solr",
> >>>>>> "base_url":"http://jamiesmac:8501/solr"},
> >>>>>> "jamiesmac:8502_solr_slice2_shard1":{
> >>>>>> "shard_id":"slice2",
> >>>>>> "leader":"true",
> >>>>>> "state":"active",
> >>>>>> "core":"slice2_shard1",
> >>>>>> "collection":"collection1",
> >>>>>> "node_name":"jamiesmac:8502_solr",
> >>>>>> "base_url":"http://jamiesmac:8502/solr"}}}}
> >>>>>>
> >>>>>> I then added some docs to the following shards using SolrJ
> >>>>>> http://localhost:8502/solr/slice2_shard1
> >>>>>> http://localhost:8502/solr/slice1_shard2
> >>>>>>
> >>>>>> I then bring back up the other cores and I don't see replication
> >>>>>> happening. Looking at the stats for each core I see that on the
> 8501
> >>>>>> instance (the instance that was off) the number of docs is 0, so
> I've
> >>>>>> clearly set something up incorrectly. Any help on this would be
> >>>>>> greatly appreciated.
> >>>>>
> >>>>> - Mark Miller
> >>>>> lucidimagination.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>
> >> - Mark Miller
> >> lucidimagination.com
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> > - Mark Miller
> > lucidimagination.com
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
--
- Mark
http://www.lucidimagination.com
Re: SolrCloud Replication Question
Posted by Jamie Johnson <je...@gmail.com>.
nothing seems that different. In regards to the states of each I'll
try to verify tonight.
This was using a version I pulled from SVN trunk yesterday morning
On Fri, Feb 10, 2012 at 6:22 PM, Mark Miller <ma...@gmail.com> wrote:
> Also, it will help if you can mention the exact version of solrcloud you are talking about in each issue - I know you have one from the old branch, and I assume a version off trunk you are playing with - so a heads up on which and if trunk, what rev or day will help in the case that I'm trying to dupe issues that have been addressed.
>
> - Mark
>
> On Feb 10, 2012, at 6:09 PM, Mark Miller wrote:
>
>> I'm trying, but so far I don't see anything. I'll have to try and mimic your setup closer it seems.
>>
>> I tried starting up 6 solr instances on different ports as 2 shards, each with a replication factor of 3.
>>
>> Then I indexed 20k documents to the cluster and verified doc counts.
>>
>> Then I shutdown all the replicas so that only one instance served each shard.
>>
>> Then I indexed 20k documents to the cluster.
>>
>> Then I started the downed nodes and verified that they where in a recovery state.
>>
>> After enough time went by I checked and verified document counts on each instance - they where as expected.
>>
>> I guess next I can try a similar experiment using multiple cores, but if you notice anything that stands out that is largely different in what you are doing, let me know.
>>
>> The cores that are behind, does it say they are down, recovering, or active in zookeeper?
>>
>> On Feb 10, 2012, at 4:48 PM, Jamie Johnson wrote:
>>
>>> Sorry for pinging this again, is more information needed on this? I
>>> can provide more details but am not sure what to provide.
>>>
>>> On Fri, Feb 10, 2012 at 10:26 AM, Jamie Johnson <je...@gmail.com> wrote:
>>>> Sorry, I shut down the full solr instance.
>>>>
>>>> On Fri, Feb 10, 2012 at 9:42 AM, Mark Miller <ma...@gmail.com> wrote:
>>>>> Can you explain a little more how you doing this? How are you bringing the cores down and then back up? Shutting down a full solr instance, unloading the core?
>>>>>
>>>>> On Feb 10, 2012, at 9:33 AM, Jamie Johnson wrote:
>>>>>
>>>>>> I know that the latest Solr Cloud doesn't use standard replication but
>>>>>> I have a question about how it appears to be working. I currently
>>>>>> have the following cluster state
>>>>>>
>>>>>> {"collection1":{
>>>>>> "slice1":{
>>>>>> "JamiesMac.local:8501_solr_slice1_shard1":{
>>>>>> "shard_id":"slice1",
>>>>>> "state":"active",
>>>>>> "core":"slice1_shard1",
>>>>>> "collection":"collection1",
>>>>>> "node_name":"JamiesMac.local:8501_solr",
>>>>>> "base_url":"http://JamiesMac.local:8501/solr"},
>>>>>> "JamiesMac.local:8502_solr_slice1_shard2":{
>>>>>> "shard_id":"slice1",
>>>>>> "state":"active",
>>>>>> "core":"slice1_shard2",
>>>>>> "collection":"collection1",
>>>>>> "node_name":"JamiesMac.local:8502_solr",
>>>>>> "base_url":"http://JamiesMac.local:8502/solr"},
>>>>>> "jamiesmac:8501_solr_slice1_shard1":{
>>>>>> "shard_id":"slice1",
>>>>>> "state":"down",
>>>>>> "core":"slice1_shard1",
>>>>>> "collection":"collection1",
>>>>>> "node_name":"jamiesmac:8501_solr",
>>>>>> "base_url":"http://jamiesmac:8501/solr"},
>>>>>> "jamiesmac:8502_solr_slice1_shard2":{
>>>>>> "shard_id":"slice1",
>>>>>> "leader":"true",
>>>>>> "state":"active",
>>>>>> "core":"slice1_shard2",
>>>>>> "collection":"collection1",
>>>>>> "node_name":"jamiesmac:8502_solr",
>>>>>> "base_url":"http://jamiesmac:8502/solr"}},
>>>>>> "slice2":{
>>>>>> "JamiesMac.local:8501_solr_slice2_shard2":{
>>>>>> "shard_id":"slice2",
>>>>>> "state":"active",
>>>>>> "core":"slice2_shard2",
>>>>>> "collection":"collection1",
>>>>>> "node_name":"JamiesMac.local:8501_solr",
>>>>>> "base_url":"http://JamiesMac.local:8501/solr"},
>>>>>> "JamiesMac.local:8502_solr_slice2_shard1":{
>>>>>> "shard_id":"slice2",
>>>>>> "state":"active",
>>>>>> "core":"slice2_shard1",
>>>>>> "collection":"collection1",
>>>>>> "node_name":"JamiesMac.local:8502_solr",
>>>>>> "base_url":"http://JamiesMac.local:8502/solr"},
>>>>>> "jamiesmac:8501_solr_slice2_shard2":{
>>>>>> "shard_id":"slice2",
>>>>>> "state":"down",
>>>>>> "core":"slice2_shard2",
>>>>>> "collection":"collection1",
>>>>>> "node_name":"jamiesmac:8501_solr",
>>>>>> "base_url":"http://jamiesmac:8501/solr"},
>>>>>> "jamiesmac:8502_solr_slice2_shard1":{
>>>>>> "shard_id":"slice2",
>>>>>> "leader":"true",
>>>>>> "state":"active",
>>>>>> "core":"slice2_shard1",
>>>>>> "collection":"collection1",
>>>>>> "node_name":"jamiesmac:8502_solr",
>>>>>> "base_url":"http://jamiesmac:8502/solr"}}}}
>>>>>>
>>>>>> I then added some docs to the following shards using SolrJ
>>>>>> http://localhost:8502/solr/slice2_shard1
>>>>>> http://localhost:8502/solr/slice1_shard2
>>>>>>
>>>>>> I then bring back up the other cores and I don't see replication
>>>>>> happening. Looking at the stats for each core I see that on the 8501
>>>>>> instance (the instance that was off) the number of docs is 0, so I've
>>>>>> clearly set something up incorrectly. Any help on this would be
>>>>>> greatly appreciated.
>>>>>
>>>>> - Mark Miller
>>>>> lucidimagination.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>
>> - Mark Miller
>> lucidimagination.com
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
Re: SolrCloud Replication Question
Posted by Mark Miller <ma...@gmail.com>.
Also, it will help if you can mention the exact version of solrcloud you are talking about in each issue - I know you have one from the old branch, and I assume a version off trunk you are playing with - so a heads up on which and if trunk, what rev or day will help in the case that I'm trying to dupe issues that have been addressed.
- Mark
On Feb 10, 2012, at 6:09 PM, Mark Miller wrote:
> I'm trying, but so far I don't see anything. I'll have to try and mimic your setup closer it seems.
>
> I tried starting up 6 solr instances on different ports as 2 shards, each with a replication factor of 3.
>
> Then I indexed 20k documents to the cluster and verified doc counts.
>
> Then I shutdown all the replicas so that only one instance served each shard.
>
> Then I indexed 20k documents to the cluster.
>
> Then I started the downed nodes and verified that they where in a recovery state.
>
> After enough time went by I checked and verified document counts on each instance - they where as expected.
>
> I guess next I can try a similar experiment using multiple cores, but if you notice anything that stands out that is largely different in what you are doing, let me know.
>
> The cores that are behind, does it say they are down, recovering, or active in zookeeper?
>
> On Feb 10, 2012, at 4:48 PM, Jamie Johnson wrote:
>
>> Sorry for pinging this again, is more information needed on this? I
>> can provide more details but am not sure what to provide.
>>
>> On Fri, Feb 10, 2012 at 10:26 AM, Jamie Johnson <je...@gmail.com> wrote:
>>> Sorry, I shut down the full solr instance.
>>>
>>> On Fri, Feb 10, 2012 at 9:42 AM, Mark Miller <ma...@gmail.com> wrote:
>>>> Can you explain a little more how you doing this? How are you bringing the cores down and then back up? Shutting down a full solr instance, unloading the core?
>>>>
>>>> On Feb 10, 2012, at 9:33 AM, Jamie Johnson wrote:
>>>>
>>>>> I know that the latest Solr Cloud doesn't use standard replication but
>>>>> I have a question about how it appears to be working. I currently
>>>>> have the following cluster state
>>>>>
>>>>> {"collection1":{
>>>>> "slice1":{
>>>>> "JamiesMac.local:8501_solr_slice1_shard1":{
>>>>> "shard_id":"slice1",
>>>>> "state":"active",
>>>>> "core":"slice1_shard1",
>>>>> "collection":"collection1",
>>>>> "node_name":"JamiesMac.local:8501_solr",
>>>>> "base_url":"http://JamiesMac.local:8501/solr"},
>>>>> "JamiesMac.local:8502_solr_slice1_shard2":{
>>>>> "shard_id":"slice1",
>>>>> "state":"active",
>>>>> "core":"slice1_shard2",
>>>>> "collection":"collection1",
>>>>> "node_name":"JamiesMac.local:8502_solr",
>>>>> "base_url":"http://JamiesMac.local:8502/solr"},
>>>>> "jamiesmac:8501_solr_slice1_shard1":{
>>>>> "shard_id":"slice1",
>>>>> "state":"down",
>>>>> "core":"slice1_shard1",
>>>>> "collection":"collection1",
>>>>> "node_name":"jamiesmac:8501_solr",
>>>>> "base_url":"http://jamiesmac:8501/solr"},
>>>>> "jamiesmac:8502_solr_slice1_shard2":{
>>>>> "shard_id":"slice1",
>>>>> "leader":"true",
>>>>> "state":"active",
>>>>> "core":"slice1_shard2",
>>>>> "collection":"collection1",
>>>>> "node_name":"jamiesmac:8502_solr",
>>>>> "base_url":"http://jamiesmac:8502/solr"}},
>>>>> "slice2":{
>>>>> "JamiesMac.local:8501_solr_slice2_shard2":{
>>>>> "shard_id":"slice2",
>>>>> "state":"active",
>>>>> "core":"slice2_shard2",
>>>>> "collection":"collection1",
>>>>> "node_name":"JamiesMac.local:8501_solr",
>>>>> "base_url":"http://JamiesMac.local:8501/solr"},
>>>>> "JamiesMac.local:8502_solr_slice2_shard1":{
>>>>> "shard_id":"slice2",
>>>>> "state":"active",
>>>>> "core":"slice2_shard1",
>>>>> "collection":"collection1",
>>>>> "node_name":"JamiesMac.local:8502_solr",
>>>>> "base_url":"http://JamiesMac.local:8502/solr"},
>>>>> "jamiesmac:8501_solr_slice2_shard2":{
>>>>> "shard_id":"slice2",
>>>>> "state":"down",
>>>>> "core":"slice2_shard2",
>>>>> "collection":"collection1",
>>>>> "node_name":"jamiesmac:8501_solr",
>>>>> "base_url":"http://jamiesmac:8501/solr"},
>>>>> "jamiesmac:8502_solr_slice2_shard1":{
>>>>> "shard_id":"slice2",
>>>>> "leader":"true",
>>>>> "state":"active",
>>>>> "core":"slice2_shard1",
>>>>> "collection":"collection1",
>>>>> "node_name":"jamiesmac:8502_solr",
>>>>> "base_url":"http://jamiesmac:8502/solr"}}}}
>>>>>
>>>>> I then added some docs to the following shards using SolrJ
>>>>> http://localhost:8502/solr/slice2_shard1
>>>>> http://localhost:8502/solr/slice1_shard2
>>>>>
>>>>> I then bring back up the other cores and I don't see replication
>>>>> happening. Looking at the stats for each core I see that on the 8501
>>>>> instance (the instance that was off) the number of docs is 0, so I've
>>>>> clearly set something up incorrectly. Any help on this would be
>>>>> greatly appreciated.
>>>>
>>>> - Mark Miller
>>>> lucidimagination.com
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
- Mark Miller
lucidimagination.com
Re: SolrCloud Replication Question
Posted by Mark Miller <ma...@gmail.com>.
I'm trying, but so far I don't see anything. I'll have to try and mimic your setup closer it seems.
I tried starting up 6 solr instances on different ports as 2 shards, each with a replication factor of 3.
Then I indexed 20k documents to the cluster and verified doc counts.
Then I shutdown all the replicas so that only one instance served each shard.
Then I indexed 20k documents to the cluster.
Then I started the downed nodes and verified that they where in a recovery state.
After enough time went by I checked and verified document counts on each instance - they where as expected.
I guess next I can try a similar experiment using multiple cores, but if you notice anything that stands out that is largely different in what you are doing, let me know.
The cores that are behind, does it say they are down, recovering, or active in zookeeper?
On Feb 10, 2012, at 4:48 PM, Jamie Johnson wrote:
> Sorry for pinging this again, is more information needed on this? I
> can provide more details but am not sure what to provide.
>
> On Fri, Feb 10, 2012 at 10:26 AM, Jamie Johnson <je...@gmail.com> wrote:
>> Sorry, I shut down the full solr instance.
>>
>> On Fri, Feb 10, 2012 at 9:42 AM, Mark Miller <ma...@gmail.com> wrote:
>>> Can you explain a little more how you doing this? How are you bringing the cores down and then back up? Shutting down a full solr instance, unloading the core?
>>>
>>> On Feb 10, 2012, at 9:33 AM, Jamie Johnson wrote:
>>>
>>>> I know that the latest Solr Cloud doesn't use standard replication but
>>>> I have a question about how it appears to be working. I currently
>>>> have the following cluster state
>>>>
>>>> {"collection1":{
>>>> "slice1":{
>>>> "JamiesMac.local:8501_solr_slice1_shard1":{
>>>> "shard_id":"slice1",
>>>> "state":"active",
>>>> "core":"slice1_shard1",
>>>> "collection":"collection1",
>>>> "node_name":"JamiesMac.local:8501_solr",
>>>> "base_url":"http://JamiesMac.local:8501/solr"},
>>>> "JamiesMac.local:8502_solr_slice1_shard2":{
>>>> "shard_id":"slice1",
>>>> "state":"active",
>>>> "core":"slice1_shard2",
>>>> "collection":"collection1",
>>>> "node_name":"JamiesMac.local:8502_solr",
>>>> "base_url":"http://JamiesMac.local:8502/solr"},
>>>> "jamiesmac:8501_solr_slice1_shard1":{
>>>> "shard_id":"slice1",
>>>> "state":"down",
>>>> "core":"slice1_shard1",
>>>> "collection":"collection1",
>>>> "node_name":"jamiesmac:8501_solr",
>>>> "base_url":"http://jamiesmac:8501/solr"},
>>>> "jamiesmac:8502_solr_slice1_shard2":{
>>>> "shard_id":"slice1",
>>>> "leader":"true",
>>>> "state":"active",
>>>> "core":"slice1_shard2",
>>>> "collection":"collection1",
>>>> "node_name":"jamiesmac:8502_solr",
>>>> "base_url":"http://jamiesmac:8502/solr"}},
>>>> "slice2":{
>>>> "JamiesMac.local:8501_solr_slice2_shard2":{
>>>> "shard_id":"slice2",
>>>> "state":"active",
>>>> "core":"slice2_shard2",
>>>> "collection":"collection1",
>>>> "node_name":"JamiesMac.local:8501_solr",
>>>> "base_url":"http://JamiesMac.local:8501/solr"},
>>>> "JamiesMac.local:8502_solr_slice2_shard1":{
>>>> "shard_id":"slice2",
>>>> "state":"active",
>>>> "core":"slice2_shard1",
>>>> "collection":"collection1",
>>>> "node_name":"JamiesMac.local:8502_solr",
>>>> "base_url":"http://JamiesMac.local:8502/solr"},
>>>> "jamiesmac:8501_solr_slice2_shard2":{
>>>> "shard_id":"slice2",
>>>> "state":"down",
>>>> "core":"slice2_shard2",
>>>> "collection":"collection1",
>>>> "node_name":"jamiesmac:8501_solr",
>>>> "base_url":"http://jamiesmac:8501/solr"},
>>>> "jamiesmac:8502_solr_slice2_shard1":{
>>>> "shard_id":"slice2",
>>>> "leader":"true",
>>>> "state":"active",
>>>> "core":"slice2_shard1",
>>>> "collection":"collection1",
>>>> "node_name":"jamiesmac:8502_solr",
>>>> "base_url":"http://jamiesmac:8502/solr"}}}}
>>>>
>>>> I then added some docs to the following shards using SolrJ
>>>> http://localhost:8502/solr/slice2_shard1
>>>> http://localhost:8502/solr/slice1_shard2
>>>>
>>>> I then bring back up the other cores and I don't see replication
>>>> happening. Looking at the stats for each core I see that on the 8501
>>>> instance (the instance that was off) the number of docs is 0, so I've
>>>> clearly set something up incorrectly. Any help on this would be
>>>> greatly appreciated.
>>>
>>> - Mark Miller
>>> lucidimagination.com
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
- Mark Miller
lucidimagination.com
Re: SolrCloud Replication Question
Posted by Jamie Johnson <je...@gmail.com>.
Sorry for pinging this again, is more information needed on this? I
can provide more details but am not sure what to provide.
On Fri, Feb 10, 2012 at 10:26 AM, Jamie Johnson <je...@gmail.com> wrote:
> Sorry, I shut down the full solr instance.
>
> On Fri, Feb 10, 2012 at 9:42 AM, Mark Miller <ma...@gmail.com> wrote:
>> Can you explain a little more how you doing this? How are you bringing the cores down and then back up? Shutting down a full solr instance, unloading the core?
>>
>> On Feb 10, 2012, at 9:33 AM, Jamie Johnson wrote:
>>
>>> I know that the latest Solr Cloud doesn't use standard replication but
>>> I have a question about how it appears to be working. I currently
>>> have the following cluster state
>>>
>>> {"collection1":{
>>> "slice1":{
>>> "JamiesMac.local:8501_solr_slice1_shard1":{
>>> "shard_id":"slice1",
>>> "state":"active",
>>> "core":"slice1_shard1",
>>> "collection":"collection1",
>>> "node_name":"JamiesMac.local:8501_solr",
>>> "base_url":"http://JamiesMac.local:8501/solr"},
>>> "JamiesMac.local:8502_solr_slice1_shard2":{
>>> "shard_id":"slice1",
>>> "state":"active",
>>> "core":"slice1_shard2",
>>> "collection":"collection1",
>>> "node_name":"JamiesMac.local:8502_solr",
>>> "base_url":"http://JamiesMac.local:8502/solr"},
>>> "jamiesmac:8501_solr_slice1_shard1":{
>>> "shard_id":"slice1",
>>> "state":"down",
>>> "core":"slice1_shard1",
>>> "collection":"collection1",
>>> "node_name":"jamiesmac:8501_solr",
>>> "base_url":"http://jamiesmac:8501/solr"},
>>> "jamiesmac:8502_solr_slice1_shard2":{
>>> "shard_id":"slice1",
>>> "leader":"true",
>>> "state":"active",
>>> "core":"slice1_shard2",
>>> "collection":"collection1",
>>> "node_name":"jamiesmac:8502_solr",
>>> "base_url":"http://jamiesmac:8502/solr"}},
>>> "slice2":{
>>> "JamiesMac.local:8501_solr_slice2_shard2":{
>>> "shard_id":"slice2",
>>> "state":"active",
>>> "core":"slice2_shard2",
>>> "collection":"collection1",
>>> "node_name":"JamiesMac.local:8501_solr",
>>> "base_url":"http://JamiesMac.local:8501/solr"},
>>> "JamiesMac.local:8502_solr_slice2_shard1":{
>>> "shard_id":"slice2",
>>> "state":"active",
>>> "core":"slice2_shard1",
>>> "collection":"collection1",
>>> "node_name":"JamiesMac.local:8502_solr",
>>> "base_url":"http://JamiesMac.local:8502/solr"},
>>> "jamiesmac:8501_solr_slice2_shard2":{
>>> "shard_id":"slice2",
>>> "state":"down",
>>> "core":"slice2_shard2",
>>> "collection":"collection1",
>>> "node_name":"jamiesmac:8501_solr",
>>> "base_url":"http://jamiesmac:8501/solr"},
>>> "jamiesmac:8502_solr_slice2_shard1":{
>>> "shard_id":"slice2",
>>> "leader":"true",
>>> "state":"active",
>>> "core":"slice2_shard1",
>>> "collection":"collection1",
>>> "node_name":"jamiesmac:8502_solr",
>>> "base_url":"http://jamiesmac:8502/solr"}}}}
>>>
>>> I then added some docs to the following shards using SolrJ
>>> http://localhost:8502/solr/slice2_shard1
>>> http://localhost:8502/solr/slice1_shard2
>>>
>>> I then bring back up the other cores and I don't see replication
>>> happening. Looking at the stats for each core I see that on the 8501
>>> instance (the instance that was off) the number of docs is 0, so I've
>>> clearly set something up incorrectly. Any help on this would be
>>> greatly appreciated.
>>
>> - Mark Miller
>> lucidimagination.com
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
Re: SolrCloud Replication Question
Posted by Jamie Johnson <je...@gmail.com>.
Sorry, I shut down the full solr instance.
On Fri, Feb 10, 2012 at 9:42 AM, Mark Miller <ma...@gmail.com> wrote:
> Can you explain a little more how you doing this? How are you bringing the cores down and then back up? Shutting down a full solr instance, unloading the core?
>
> On Feb 10, 2012, at 9:33 AM, Jamie Johnson wrote:
>
>> I know that the latest Solr Cloud doesn't use standard replication but
>> I have a question about how it appears to be working. I currently
>> have the following cluster state
>>
>> {"collection1":{
>> "slice1":{
>> "JamiesMac.local:8501_solr_slice1_shard1":{
>> "shard_id":"slice1",
>> "state":"active",
>> "core":"slice1_shard1",
>> "collection":"collection1",
>> "node_name":"JamiesMac.local:8501_solr",
>> "base_url":"http://JamiesMac.local:8501/solr"},
>> "JamiesMac.local:8502_solr_slice1_shard2":{
>> "shard_id":"slice1",
>> "state":"active",
>> "core":"slice1_shard2",
>> "collection":"collection1",
>> "node_name":"JamiesMac.local:8502_solr",
>> "base_url":"http://JamiesMac.local:8502/solr"},
>> "jamiesmac:8501_solr_slice1_shard1":{
>> "shard_id":"slice1",
>> "state":"down",
>> "core":"slice1_shard1",
>> "collection":"collection1",
>> "node_name":"jamiesmac:8501_solr",
>> "base_url":"http://jamiesmac:8501/solr"},
>> "jamiesmac:8502_solr_slice1_shard2":{
>> "shard_id":"slice1",
>> "leader":"true",
>> "state":"active",
>> "core":"slice1_shard2",
>> "collection":"collection1",
>> "node_name":"jamiesmac:8502_solr",
>> "base_url":"http://jamiesmac:8502/solr"}},
>> "slice2":{
>> "JamiesMac.local:8501_solr_slice2_shard2":{
>> "shard_id":"slice2",
>> "state":"active",
>> "core":"slice2_shard2",
>> "collection":"collection1",
>> "node_name":"JamiesMac.local:8501_solr",
>> "base_url":"http://JamiesMac.local:8501/solr"},
>> "JamiesMac.local:8502_solr_slice2_shard1":{
>> "shard_id":"slice2",
>> "state":"active",
>> "core":"slice2_shard1",
>> "collection":"collection1",
>> "node_name":"JamiesMac.local:8502_solr",
>> "base_url":"http://JamiesMac.local:8502/solr"},
>> "jamiesmac:8501_solr_slice2_shard2":{
>> "shard_id":"slice2",
>> "state":"down",
>> "core":"slice2_shard2",
>> "collection":"collection1",
>> "node_name":"jamiesmac:8501_solr",
>> "base_url":"http://jamiesmac:8501/solr"},
>> "jamiesmac:8502_solr_slice2_shard1":{
>> "shard_id":"slice2",
>> "leader":"true",
>> "state":"active",
>> "core":"slice2_shard1",
>> "collection":"collection1",
>> "node_name":"jamiesmac:8502_solr",
>> "base_url":"http://jamiesmac:8502/solr"}}}}
>>
>> I then added some docs to the following shards using SolrJ
>> http://localhost:8502/solr/slice2_shard1
>> http://localhost:8502/solr/slice1_shard2
>>
>> I then bring back up the other cores and I don't see replication
>> happening. Looking at the stats for each core I see that on the 8501
>> instance (the instance that was off) the number of docs is 0, so I've
>> clearly set something up incorrectly. Any help on this would be
>> greatly appreciated.
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
Re: SolrCloud Replication Question
Posted by Mark Miller <ma...@gmail.com>.
Can you explain a little more how you doing this? How are you bringing the cores down and then back up? Shutting down a full solr instance, unloading the core?
On Feb 10, 2012, at 9:33 AM, Jamie Johnson wrote:
> I know that the latest Solr Cloud doesn't use standard replication but
> I have a question about how it appears to be working. I currently
> have the following cluster state
>
> {"collection1":{
> "slice1":{
> "JamiesMac.local:8501_solr_slice1_shard1":{
> "shard_id":"slice1",
> "state":"active",
> "core":"slice1_shard1",
> "collection":"collection1",
> "node_name":"JamiesMac.local:8501_solr",
> "base_url":"http://JamiesMac.local:8501/solr"},
> "JamiesMac.local:8502_solr_slice1_shard2":{
> "shard_id":"slice1",
> "state":"active",
> "core":"slice1_shard2",
> "collection":"collection1",
> "node_name":"JamiesMac.local:8502_solr",
> "base_url":"http://JamiesMac.local:8502/solr"},
> "jamiesmac:8501_solr_slice1_shard1":{
> "shard_id":"slice1",
> "state":"down",
> "core":"slice1_shard1",
> "collection":"collection1",
> "node_name":"jamiesmac:8501_solr",
> "base_url":"http://jamiesmac:8501/solr"},
> "jamiesmac:8502_solr_slice1_shard2":{
> "shard_id":"slice1",
> "leader":"true",
> "state":"active",
> "core":"slice1_shard2",
> "collection":"collection1",
> "node_name":"jamiesmac:8502_solr",
> "base_url":"http://jamiesmac:8502/solr"}},
> "slice2":{
> "JamiesMac.local:8501_solr_slice2_shard2":{
> "shard_id":"slice2",
> "state":"active",
> "core":"slice2_shard2",
> "collection":"collection1",
> "node_name":"JamiesMac.local:8501_solr",
> "base_url":"http://JamiesMac.local:8501/solr"},
> "JamiesMac.local:8502_solr_slice2_shard1":{
> "shard_id":"slice2",
> "state":"active",
> "core":"slice2_shard1",
> "collection":"collection1",
> "node_name":"JamiesMac.local:8502_solr",
> "base_url":"http://JamiesMac.local:8502/solr"},
> "jamiesmac:8501_solr_slice2_shard2":{
> "shard_id":"slice2",
> "state":"down",
> "core":"slice2_shard2",
> "collection":"collection1",
> "node_name":"jamiesmac:8501_solr",
> "base_url":"http://jamiesmac:8501/solr"},
> "jamiesmac:8502_solr_slice2_shard1":{
> "shard_id":"slice2",
> "leader":"true",
> "state":"active",
> "core":"slice2_shard1",
> "collection":"collection1",
> "node_name":"jamiesmac:8502_solr",
> "base_url":"http://jamiesmac:8502/solr"}}}}
>
> I then added some docs to the following shards using SolrJ
> http://localhost:8502/solr/slice2_shard1
> http://localhost:8502/solr/slice1_shard2
>
> I then bring back up the other cores and I don't see replication
> happening. Looking at the stats for each core I see that on the 8501
> instance (the instance that was off) the number of docs is 0, so I've
> clearly set something up incorrectly. Any help on this would be
> greatly appreciated.
- Mark Miller
lucidimagination.com
Re: SolrCloud Replication Question
Posted by Jamie Johnson <je...@gmail.com>.
I don't see any errors in the log. here are the following scripts I'm
running, and to create the cores I run the following commands
curl 'http://localhost:8501/solr/admin/cores?action=CREATE&name=slice1_shard1&collection=collection1&shard=slice1&collection.configName=config1'
curl 'http://localhost:8501/solr/admin/cores?action=CREATE&name=slice2_shard2&collection=collection1&shard=slice2&collection.configName=config1'
curl 'http://localhost:8502/solr/admin/cores?action=CREATE&name=slice2_shard1&collection=collection1&shard=slice2&collection.configName=config1'
curl 'http://localhost:8502/solr/admin/cores?action=CREATE&name=slice1_shard2&collection=collection1&shard=slice1&collection.configName=config1'
after doing this the nodes are immediately marked as down in
clusterstate.json. Restating the solr instances I see that which ever
I start first shows up as active, and the other is down. There are no
errors in the logs either.
On Sat, Feb 11, 2012 at 9:48 PM, Mark Miller <ma...@gmail.com> wrote:
> Yeah, that is what I would expect - for a node to be marked as down, it either didn't finish starting, or it gave up recovering...either case should be logged. You might try searching for the recover keyword and see if there are any interesting bits around that.
>
> Meanwhile, I have dug up a couple issues around recovery and committed fixes to trunk - still playing around...
>
> On Feb 11, 2012, at 8:44 PM, Jamie Johnson wrote:
>
>> I didn't see anything in the logs, would it be an error?
>>
>> On Sat, Feb 11, 2012 at 3:58 PM, Mark Miller <ma...@gmail.com> wrote:
>>>
>>> On Feb 11, 2012, at 3:08 PM, Jamie Johnson wrote:
>>>
>>>> I wiped the zk and started over (when I switch networks I get
>>>> different host names and honestly haven't dug into why). That being
>>>> said the latest state shows all in sync, why would the cores show up
>>>> as down?
>>>
>>>
>>> If recovery fails X times (say because the leader can't be reached from the replica), a node is marked as down. It can't be active, and technically it has stopped trying to recover (it tries X times and eventually give up until you restart it).
>>>
>>> Side note, I recently ran into this issue: SOLR-3122 - fix coming soon. Not sure if you have looked at your logs or not, but perhaps it's involved.
>>>
>>> - Mark Miller
>>> lucidimagination.com
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
Re: SolrCloud Replication Question
Posted by Mark Miller <ma...@gmail.com>.
Yeah, that is what I would expect - for a node to be marked as down, it either didn't finish starting, or it gave up recovering...either case should be logged. You might try searching for the recover keyword and see if there are any interesting bits around that.
Meanwhile, I have dug up a couple issues around recovery and committed fixes to trunk - still playing around...
On Feb 11, 2012, at 8:44 PM, Jamie Johnson wrote:
> I didn't see anything in the logs, would it be an error?
>
> On Sat, Feb 11, 2012 at 3:58 PM, Mark Miller <ma...@gmail.com> wrote:
>>
>> On Feb 11, 2012, at 3:08 PM, Jamie Johnson wrote:
>>
>>> I wiped the zk and started over (when I switch networks I get
>>> different host names and honestly haven't dug into why). That being
>>> said the latest state shows all in sync, why would the cores show up
>>> as down?
>>
>>
>> If recovery fails X times (say because the leader can't be reached from the replica), a node is marked as down. It can't be active, and technically it has stopped trying to recover (it tries X times and eventually give up until you restart it).
>>
>> Side note, I recently ran into this issue: SOLR-3122 - fix coming soon. Not sure if you have looked at your logs or not, but perhaps it's involved.
>>
>> - Mark Miller
>> lucidimagination.com
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
- Mark Miller
lucidimagination.com
Re: SolrCloud Replication Question
Posted by Jamie Johnson <je...@gmail.com>.
I didn't see anything in the logs, would it be an error?
On Sat, Feb 11, 2012 at 3:58 PM, Mark Miller <ma...@gmail.com> wrote:
>
> On Feb 11, 2012, at 3:08 PM, Jamie Johnson wrote:
>
>> I wiped the zk and started over (when I switch networks I get
>> different host names and honestly haven't dug into why). That being
>> said the latest state shows all in sync, why would the cores show up
>> as down?
>
>
> If recovery fails X times (say because the leader can't be reached from the replica), a node is marked as down. It can't be active, and technically it has stopped trying to recover (it tries X times and eventually give up until you restart it).
>
> Side note, I recently ran into this issue: SOLR-3122 - fix coming soon. Not sure if you have looked at your logs or not, but perhaps it's involved.
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
Re: SolrCloud Replication Question
Posted by Mark Miller <ma...@gmail.com>.
On Feb 11, 2012, at 3:08 PM, Jamie Johnson wrote:
> I wiped the zk and started over (when I switch networks I get
> different host names and honestly haven't dug into why). That being
> said the latest state shows all in sync, why would the cores show up
> as down?
If recovery fails X times (say because the leader can't be reached from the replica), a node is marked as down. It can't be active, and technically it has stopped trying to recover (it tries X times and eventually give up until you restart it).
Side note, I recently ran into this issue: SOLR-3122 - fix coming soon. Not sure if you have looked at your logs or not, but perhaps it's involved.
- Mark Miller
lucidimagination.com
Re: SolrCloud Replication Question
Posted by Jamie Johnson <je...@gmail.com>.
I wiped the zk and started over (when I switch networks I get
different host names and honestly haven't dug into why). That being
said the latest state shows all in sync, why would the cores show up
as down?
On Sat, Feb 11, 2012 at 11:08 AM, Mark Miller <ma...@gmail.com> wrote:
>
> On Feb 10, 2012, at 9:40 PM, Jamie Johnson wrote:
>
>>
>>
>> how'd you resolve this issue?
>>
>
>
> I was basing my guess on seeing "JamiesMac.local" and "jamiesmac" in your first cluster state dump - your latest doesn't seem to mismatch like that though.
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
Re: SolrCloud Replication Question
Posted by Mark Miller <ma...@gmail.com>.
On Feb 10, 2012, at 9:40 PM, Jamie Johnson wrote:
>
>
> how'd you resolve this issue?
>
I was basing my guess on seeing "JamiesMac.local" and "jamiesmac" in your first cluster state dump - your latest doesn't seem to mismatch like that though.
- Mark Miller
lucidimagination.com
Re: SolrCloud Replication Question
Posted by Jamie Johnson <je...@gmail.com>.
hmm....perhaps I'm seeing the issue you're speaking of. I have
everything running right now and my state is as follows:
{"collection1":{
"slice1":{
"JamiesMac.local:8501_solr_slice1_shard1":{
"shard_id":"slice1",
"leader":"true",
"state":"active",
"core":"slice1_shard1",
"collection":"collection1",
"node_name":"JamiesMac.local:8501_solr",
"base_url":"http://JamiesMac.local:8501/solr"},
"JamiesMac.local:8502_solr_slice1_shard2":{
"shard_id":"slice1",
"state":"down",
"core":"slice1_shard2",
"collection":"collection1",
"node_name":"JamiesMac.local:8502_solr",
"base_url":"http://JamiesMac.local:8502/solr"}},
"slice2":{
"JamiesMac.local:8502_solr_slice2_shard1":{
"shard_id":"slice2",
"leader":"true",
"state":"active",
"core":"slice2_shard1",
"collection":"collection1",
"node_name":"JamiesMac.local:8502_solr",
"base_url":"http://JamiesMac.local:8502/solr"},
"JamiesMac.local:8501_solr_slice2_shard2":{
"shard_id":"slice2",
"state":"down",
"core":"slice2_shard2",
"collection":"dataspace",
"node_name":"JamiesMac.local:8501_solr",
"base_url":"http://JamiesMac.local:8501/solr"}}}}
how'd you resolve this issue?
On Fri, Feb 10, 2012 at 8:49 PM, Mark Miller <ma...@gmail.com> wrote:
>
> On Feb 10, 2012, at 9:33 AM, Jamie Johnson wrote:
>
>> jamiesmac
>
> Another note:
>
> Have no idea if this is involved, but when I do tests with my linux box and mac I run into the following:
>
> My linux box auto finds the address of halfmetal and my macbook mbpro.local. If I accept those defaults, my mac connect reach my linux box. It can only reach the linux box through halfmetal.local, and so I have to override the host on the linux box to advertise as halfmetal.local and then they can talk.
>
> In the bad case, if my leaders where on the linux box, they would be able to forward to the mac no problem, but then if shards on the mac needed to recover, they would fail to reach the linux box through the halfmetal address.
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
Re: SolrCloud Replication Question
Posted by Mark Miller <ma...@gmail.com>.
On Feb 10, 2012, at 9:33 AM, Jamie Johnson wrote:
> jamiesmac
Another note:
Have no idea if this is involved, but when I do tests with my linux box and mac I run into the following:
My linux box auto finds the address of halfmetal and my macbook mbpro.local. If I accept those defaults, my mac connect reach my linux box. It can only reach the linux box through halfmetal.local, and so I have to override the host on the linux box to advertise as halfmetal.local and then they can talk.
In the bad case, if my leaders where on the linux box, they would be able to forward to the mac no problem, but then if shards on the mac needed to recover, they would fail to reach the linux box through the halfmetal address.
- Mark Miller
lucidimagination.com