You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Arnon Yogev <AR...@il.ibm.com> on 2015/06/23 14:19:42 UTC
Implicit Router Configurations
We have a use case where documents are indexed in shards according to a
specific field (shard per user), and the number of shards is unknown when
creating the collection.
For that purpose we use the implicit router and define router.field=user.
>From what we've seen, the only way to define an implicit router is during
the collection creation.
Moreover, the router definitions (router.name and router.field) are kept
only in clusterstate.json and not in any solr configuration file on disk.
In some cases solr state becomes inconsistent and we need to delete the
configs from ZK and restart the solr server. The behavior we see is the
new clusterstate.json generated by solr on startup has the default
router.name=compositeId, which is not what we defined during creation.
Are we missing something? Is there a place to configure the implicit
router on disk such that it will be persistent?
Thanks,
Arnon
Re: Implicit Router Configurations
Posted by Erick Erickson <er...@gmail.com>.
There's also a nifty plugin for IntelliJ that'll allow you to edit files on ZK
if you use that IDE.
On Wed, Jun 24, 2015 at 4:51 AM, Upayavira <uv...@odoko.co.uk> wrote:
> You can use the server/scripts/cloud-scripts/zkcli.sh script (or the cmd
> one) in server/scripts/cloud-scripts. Note, in older versions this is in
> example/scripts/cloud-scripts.
>
> I just used this command to get the file from zookeeper:
>
> server/scripts/cloud-scripts/zkcli.sh -z localhost:9983 -cmd getfile
> /clusterstate.json clusterstate.json
>
> You can use -cmd putfile to push it back to Zookeeper. As Erick says,
> have all nodes on your cluster down at the time. And as Erick says, this
> is not something that people are recommended to be doing generally.
>
> Upayavira
>
> On Wed, Jun 24, 2015, at 07:54 AM, Arnon Yogev wrote:
>> Thank you Erick,
>>
>> What is the recommended way to manually change clusterstate.json?
>> Is there a java code \ script way of editing a file in ZK?
>>
>> Best,
>> Arnon
>>
>>
>>
>> From: Erick Erickson <er...@gmail.com>
>> To: solr-user@lucene.apache.org
>> Date: 23/06/2015 09:09 PM
>> Subject: Re: Implicit Router Configurations
>>
>>
>>
>> Please raise a JIRA for this, I can see why this would occur.
>> You can manually change the clusterstate.json file when this
>> happens as a stop-gap, I'd have all the Solr instances down
>> when doing this though.
>>
>> Best,
>> Erick
>>
>>
>>
>> On Tue, Jun 23, 2015 at 8:19 AM, Arnon Yogev <AR...@il.ibm.com> wrote:
>> > We have a use case where documents are indexed in shards according to a
>> > specific field (shard per user), and the number of shards is unknown
>> when
>> > creating the collection.
>> > For that purpose we use the implicit router and define
>> router.field=user.
>> >
>> > From what we've seen, the only way to define an implicit router is
>> during
>> > the collection creation.
>> > Moreover, the router definitions (router.name and router.field) are kept
>> > only in clusterstate.json and not in any solr configuration file on
>> disk.
>> >
>> > In some cases solr state becomes inconsistent and we need to delete the
>> > configs from ZK and restart the solr server. The behavior we see is the
>> > new clusterstate.json generated by solr on startup has the default
>> > router.name=compositeId, which is not what we defined during creation.
>> >
>> > Are we missing something? Is there a place to configure the implicit
>> > router on disk such that it will be persistent?
>> >
>> > Thanks,
>> > Arnon
>> >
>> >
>>
>>
Re: Implicit Router Configurations
Posted by Upayavira <uv...@odoko.co.uk>.
You can use the server/scripts/cloud-scripts/zkcli.sh script (or the cmd
one) in server/scripts/cloud-scripts. Note, in older versions this is in
example/scripts/cloud-scripts.
I just used this command to get the file from zookeeper:
server/scripts/cloud-scripts/zkcli.sh -z localhost:9983 -cmd getfile
/clusterstate.json clusterstate.json
You can use -cmd putfile to push it back to Zookeeper. As Erick says,
have all nodes on your cluster down at the time. And as Erick says, this
is not something that people are recommended to be doing generally.
Upayavira
On Wed, Jun 24, 2015, at 07:54 AM, Arnon Yogev wrote:
> Thank you Erick,
>
> What is the recommended way to manually change clusterstate.json?
> Is there a java code \ script way of editing a file in ZK?
>
> Best,
> Arnon
>
>
>
> From: Erick Erickson <er...@gmail.com>
> To: solr-user@lucene.apache.org
> Date: 23/06/2015 09:09 PM
> Subject: Re: Implicit Router Configurations
>
>
>
> Please raise a JIRA for this, I can see why this would occur.
> You can manually change the clusterstate.json file when this
> happens as a stop-gap, I'd have all the Solr instances down
> when doing this though.
>
> Best,
> Erick
>
>
>
> On Tue, Jun 23, 2015 at 8:19 AM, Arnon Yogev <AR...@il.ibm.com> wrote:
> > We have a use case where documents are indexed in shards according to a
> > specific field (shard per user), and the number of shards is unknown
> when
> > creating the collection.
> > For that purpose we use the implicit router and define
> router.field=user.
> >
> > From what we've seen, the only way to define an implicit router is
> during
> > the collection creation.
> > Moreover, the router definitions (router.name and router.field) are kept
> > only in clusterstate.json and not in any solr configuration file on
> disk.
> >
> > In some cases solr state becomes inconsistent and we need to delete the
> > configs from ZK and restart the solr server. The behavior we see is the
> > new clusterstate.json generated by solr on startup has the default
> > router.name=compositeId, which is not what we defined during creation.
> >
> > Are we missing something? Is there a place to configure the implicit
> > router on disk such that it will be persistent?
> >
> > Thanks,
> > Arnon
> >
> >
>
>
Re: Implicit Router Configurations
Posted by Arnon Yogev <AR...@il.ibm.com>.
Thank you Erick,
What is the recommended way to manually change clusterstate.json?
Is there a java code \ script way of editing a file in ZK?
Best,
Arnon
From: Erick Erickson <er...@gmail.com>
To: solr-user@lucene.apache.org
Date: 23/06/2015 09:09 PM
Subject: Re: Implicit Router Configurations
Please raise a JIRA for this, I can see why this would occur.
You can manually change the clusterstate.json file when this
happens as a stop-gap, I'd have all the Solr instances down
when doing this though.
Best,
Erick
On Tue, Jun 23, 2015 at 8:19 AM, Arnon Yogev <AR...@il.ibm.com> wrote:
> We have a use case where documents are indexed in shards according to a
> specific field (shard per user), and the number of shards is unknown
when
> creating the collection.
> For that purpose we use the implicit router and define
router.field=user.
>
> From what we've seen, the only way to define an implicit router is
during
> the collection creation.
> Moreover, the router definitions (router.name and router.field) are kept
> only in clusterstate.json and not in any solr configuration file on
disk.
>
> In some cases solr state becomes inconsistent and we need to delete the
> configs from ZK and restart the solr server. The behavior we see is the
> new clusterstate.json generated by solr on startup has the default
> router.name=compositeId, which is not what we defined during creation.
>
> Are we missing something? Is there a place to configure the implicit
> router on disk such that it will be persistent?
>
> Thanks,
> Arnon
>
>
Re: Implicit Router Configurations
Posted by Erick Erickson <er...@gmail.com>.
Please raise a JIRA for this, I can see why this would occur.
You can manually change the clusterstate.json file when this
happens as a stop-gap, I'd have all the Solr instances down
when doing this though.
Best,
Erick
On Tue, Jun 23, 2015 at 8:19 AM, Arnon Yogev <AR...@il.ibm.com> wrote:
> We have a use case where documents are indexed in shards according to a
> specific field (shard per user), and the number of shards is unknown when
> creating the collection.
> For that purpose we use the implicit router and define router.field=user.
>
> From what we've seen, the only way to define an implicit router is during
> the collection creation.
> Moreover, the router definitions (router.name and router.field) are kept
> only in clusterstate.json and not in any solr configuration file on disk.
>
> In some cases solr state becomes inconsistent and we need to delete the
> configs from ZK and restart the solr server. The behavior we see is the
> new clusterstate.json generated by solr on startup has the default
> router.name=compositeId, which is not what we defined during creation.
>
> Are we missing something? Is there a place to configure the implicit
> router on disk such that it will be persistent?
>
> Thanks,
> Arnon
>
>