You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jeff Courtade <co...@gmail.com> on 2015/08/26 04:59:34 UTC
splitting shards on 4.7.2 with custom plugins
I am getting failures when trying too split shards on solr 4.2.7 with
custom plugins.
It fails regularily it cannot find the jar files for plugins when creating
the new cores/shards.
Ideas?
--
Thanks,
Jeff Courtade
M: 240.507.6116
Re: splitting shards on 4.7.2 with custom plugins
Posted by Jeff Courtade <co...@gmail.com>.
im looking at the clusterstate.json t see why it is doing this I really
dont understand it though...
{"collection1":{
"shards":{
"shard1":{
"range":"80000000-ffffffff",
"state":"active",
"replicas":{
"core_node1":{
"state":"active",
"base_url":"http://10.135.2.153:8981/solr",
"core":"collection1",
"node_name":"10.135.2.153:8981_solr",
"leader":"true"},
"core_node10":{
"state":"active",
"base_url":"http://10.135.2.153:8982/solr",
"core":"collection1",
"node_name":"10.135.2.153:8982_solr"}}},
"shard2":{
"range":"0-7fffffff",
"state":"inactive",
"replicas":{
"core_node9":{
"state":"active",
"base_url":"http://10.135.2.153:8984/solr",
"core":"collection1",
"node_name":"10.135.2.153:8984_solr",
"leader":"true"},
"core_node11":{
"state":"active",
"base_url":"http://10.135.2.153:8983/solr",
"core":"collection1",
"node_name":"10.135.2.153:8983_solr"}}},
"shard1_1":{
"range":null,
"state":"active",
"parent":null,
"replicas":{
"core_node6":{
"state":"active",
"base_url":"http://10.135.2.153:8981/solr",
"core":"collection1_shard1_1_replica1",
"node_name":"10.135.2.153:8981_solr",
"leader":"true"},
"core_node8":{
"state":"active",
"base_url":"http://10.135.2.153:8984/solr",
"core":"collection1_shard1_1_replica2",
"node_name":"10.135.2.153:8984_solr"}}},
"shard1_0":{
"range":null,
"state":"active",
"parent":null,
"replicas":{
"core_node5":{
"state":"active",
"base_url":"http://10.135.2.153:8981/solr",
"core":"collection1_shard1_0_replica1",
"node_name":"10.135.2.153:8981_solr",
"leader":"true"},
"core_node7":{
"state":"active",
"base_url":"http://10.135.2.153:8983/solr",
"core":"collection1_shard1_0_replica2",
"node_name":"10.135.2.153:8983_solr"}}},
"shard2_0":{
"range":"0-3fffffff",
"state":"active",
"replicas":{
"core_node13":{
"state":"active",
"base_url":"http://10.135.2.153:8984/solr",
"core":"collection1_shard2_0_replica1",
"node_name":"10.135.2.153:8984_solr",
"leader":"true"},
"core_node14":{
"state":"active",
"base_url":"http://10.135.2.153:8982/solr",
"core":"collection1_shard2_0_replica2",
"node_name":"10.135.2.153:8982_solr"}}},
"shard2_1":{
"range":"40000000-7fffffff",
"state":"active",
"replicas":{
"core_node12":{
"state":"active",
"base_url":"http://10.135.2.153:8984/solr",
"core":"collection1_shard2_1_replica1",
"node_name":"10.135.2.153:8984_solr",
"leader":"true"},
"core_node15":{
"state":"active",
"base_url":"http://10.135.2.153:8981/solr",
"core":"collection1_shard2_1_replica2",
"node_name":"10.135.2.153:8981_solr"}}}},
"maxShardsPerNode":"1",
"router":{"name":"compositeId"},
"replicationFactor":"1",
"autoCreated":"true"}}
--
Thanks,
Jeff Courtade
M: 240.507.6116
On Wed, Aug 26, 2015 at 8:44 AM, Jeff Courtade <co...@gmail.com>
wrote:
> Hi,
>
>
> So i got the shards too split. But they are very unbalanced.
>
>
> 7204922 total docs on the original collection
>
> shard1_0 numdocs 3661699
>
> shard1_1 numdocs 3543132
>
> shard2_0 numdocs 0
>
> shard2_1 numdcs 0
>
> Any ideas?
>
> This is what i had to do to get this to split with the custom libs
>
> I got shard1 to split successfully and it created replicas on the other
> servers in the cloud for the new shard/shards.
>
>
> This is the jist of it.
>
>
> When you split a shard solr creates a 2 new cores.
>
> When creating a core it uses the solr/solr.xml settings for classpath
> etc....
>
> This is why searches etc work fine and can find the opa plugins but when
> we called shardsplit it could not.
>
>
> I had to move the custom jars outside of the collection directory and add
> this to solr/solr.xml on the 4 nodes.
>
>
> info here .... https://wiki.apache.org/solr/Solr.xml%204.4%20and%20beyond
>
>
>
> <solr>
>
>
> <str name="sharedLib">${sharedLib:../lib}</str>
>
>
> when you restart you can see it in the log loading the jars form the new
> location.
>
>
>
> INFO - 2015-08-25 23:40:52.297; org.apache.solr.core.CoreContainer;
> loading shared library: /opt/solr/solr-4.7.2/solr01/solr/../lib
>
> INFO - 2015-08-25 23:40:52.298; org.apache.solr.core.SolrResourceLoader;
> Adding 'file:/opt/solr/solr-4.7.2/solr01/lib/commons-pool-1.6.jar' to
> classloader
>
> INFO - 2015-08-25 23:40:52.298; org.apache.solr.core.SolrResourceLoader;
> Adding
> 'file:/opt/solr/solr-4.7.2/solr01/lib/query-processing-language-0.2-SNAPSHOT.jar'
> to classloader
>
> INFO - 2015-08-25 23:40:52.299; org.apache.solr.core.SolrResourceLoader;
> Adding
> 'file:/opt/solr/solr-4.7.2/solr01/lib/jetty-continuation-8.1.10.v20130312.jar'
> to classloader
>
> INFO - 2015-08-25 23:40:52.301; org.apache.solr.core.SolrResourceLoader;
> Adding 'file:/opt/solr/solr-4.7.2/solr01/lib/groovy-all-2.0.4.jar' to
> classloader
>
> INFO - 2015-08-25 23:40:52.302; org.apache.solr.core.SolrResourceLoader;
> Adding 'file:/opt/solr/solr-4.7.2/solr01/lib/qpl-solr472-0.2-SNAPSHOT.jar'
> to classloader
>
> INFO - 2015-08-25 23:40:52.302; org.apache.solr.core.SolrResourceLoader;
> Adding
> 'file:/opt/solr/solr-4.7.2/solr01/lib/jetty-jmx-8.1.10.v20130312.jar' to
> classloader
>
> INFO - 2015-08-25 23:40:52.303; org.apache.solr.core.SolrResourceLoader;
> Adding
> 'file:/opt/solr/solr-4.7.2/solr01/lib/jetty-deploy-8.1.10.v20130312.jar' to
> classloader
>
> INFO - 2015-08-25 23:40:52.303; org.apache.solr.core.SolrResourceLoader;
> Adding 'file:/opt/solr/solr-4.7.2/solr01/lib/ext/' to classloader
>
> INFO - 2015-08-25 23:40:52.303; org.apache.solr.core.SolrResourceLoader;
> Adding
> 'file:/opt/solr/solr-4.7.2/solr01/lib/jetty-xml-8.1.10.v20130312.jar' to
> classloader
>
> so I then ran the split and checked on it in the morning
>
>
> http://dj01.aws.narasearch.us:8981/solr/admin/collections?action=SPLITSHARD&collection=collection1&shard=shard1
>
>
> it succeeded and created replicas.
>
> ls /opt/solr/solr-4.7.2/solr0*/solr/
>
> /opt/solr/solr-4.7.2/solr01/solr/:
> bin collection1_shard1_0_replica1 README.txt zoo.cfg
> collection1 collection1_shard1_1_replica1 solr.xml
>
> /opt/solr/solr-4.7.2/solr02/solr/:
> bin collection1 README.txt solr.xml zoo.cfg
>
> /opt/solr/solr-4.7.2/solr03/solr/:
> bin collection1 collection1_shard1_0_replica2 README.txt solr.xml
> zoo.cfg
>
> /opt/solr/solr-4.7.2/solr04/solr/:
> bin collection1 collection1_shard1_1_replica2 README.txt solr.xml
> zoo.cfg
>
>
> and it actually distributed it
>
> [root@dj01 solr]# du -sh *
> 4.0K bin
> 41G collection1
> 18G collection1_shard1_0_replica1
> 16G collection1_shard1_1_replica1
> 4.0K README.txt
> 4.0K solr.xml
> 4.0K zoo.cfg
> [root@dj01 solr]# du -sh
> /opt/solr/solr-4.7.2/solr04/solr/collection1_shard1_1_replica2
> 16G /opt/solr/solr-4.7.2/solr04/solr/collection1_shard1_1_replica2
> [root@dj01 solr]# du -sh
> /opt/solr/solr-4.7.2/solr03/solr/collection1_shard1_0_replica2
> 18G /opt/solr/solr-4.7.2/solr03/solr/collection1_shard1_0_replica2
>
>
> Jeff Courtade
> M: 240.507.6116
> On Aug 25, 2015 11:09 PM, "Anshum Gupta" <an...@anshumgupta.net> wrote:
>
>> Can you elaborate a bit more on the setup, what do the custom plugins do,
>> what error do you get ? It seems like a classloader/classpath issue to me
>> which doesn't really relate to Shard splitting.
>>
>>
>> On Tue, Aug 25, 2015 at 7:59 PM, Jeff Courtade <co...@gmail.com>
>> wrote:
>>
>> > I am getting failures when trying too split shards on solr 4.2.7 with
>> > custom plugins.
>> >
>> > It fails regularily it cannot find the jar files for plugins when
>> creating
>> > the new cores/shards.
>> >
>> > Ideas?
>> >
>> > --
>> > Thanks,
>> >
>> > Jeff Courtade
>> > M: 240.507.6116
>> >
>>
>>
>>
>> --
>> Anshum Gupta
>>
>
Re: splitting shards on 4.7.2 with custom plugins
Posted by Jeff Courtade <co...@gmail.com>.
Hi,
So i got the shards too split. But they are very unbalanced.
7204922 total docs on the original collection
shard1_0 numdocs 3661699
shard1_1 numdocs 3543132
shard2_0 numdocs 0
shard2_1 numdcs 0
Any ideas?
This is what i had to do to get this to split with the custom libs
I got shard1 to split successfully and it created replicas on the other
servers in the cloud for the new shard/shards.
This is the jist of it.
When you split a shard solr creates a 2 new cores.
When creating a core it uses the solr/solr.xml settings for classpath
etc....
This is why searches etc work fine and can find the opa plugins but when we
called shardsplit it could not.
I had to move the custom jars outside of the collection directory and add
this to solr/solr.xml on the 4 nodes.
info here .... https://wiki.apache.org/solr/Solr.xml%204.4%20and%20beyond
<solr>
<str name="sharedLib">${sharedLib:../lib}</str>
when you restart you can see it in the log loading the jars form the new
location.
INFO - 2015-08-25 23:40:52.297; org.apache.solr.core.CoreContainer;
loading shared library: /opt/solr/solr-4.7.2/solr01/solr/../lib
INFO - 2015-08-25 23:40:52.298; org.apache.solr.core.SolrResourceLoader;
Adding 'file:/opt/solr/solr-4.7.2/solr01/lib/commons-pool-1.6.jar' to
classloader
INFO - 2015-08-25 23:40:52.298; org.apache.solr.core.SolrResourceLoader;
Adding
'file:/opt/solr/solr-4.7.2/solr01/lib/query-processing-language-0.2-SNAPSHOT.jar'
to classloader
INFO - 2015-08-25 23:40:52.299; org.apache.solr.core.SolrResourceLoader;
Adding
'file:/opt/solr/solr-4.7.2/solr01/lib/jetty-continuation-8.1.10.v20130312.jar'
to classloader
INFO - 2015-08-25 23:40:52.301; org.apache.solr.core.SolrResourceLoader;
Adding 'file:/opt/solr/solr-4.7.2/solr01/lib/groovy-all-2.0.4.jar' to
classloader
INFO - 2015-08-25 23:40:52.302; org.apache.solr.core.SolrResourceLoader;
Adding 'file:/opt/solr/solr-4.7.2/solr01/lib/qpl-solr472-0.2-SNAPSHOT.jar'
to classloader
INFO - 2015-08-25 23:40:52.302; org.apache.solr.core.SolrResourceLoader;
Adding
'file:/opt/solr/solr-4.7.2/solr01/lib/jetty-jmx-8.1.10.v20130312.jar' to
classloader
INFO - 2015-08-25 23:40:52.303; org.apache.solr.core.SolrResourceLoader;
Adding
'file:/opt/solr/solr-4.7.2/solr01/lib/jetty-deploy-8.1.10.v20130312.jar' to
classloader
INFO - 2015-08-25 23:40:52.303; org.apache.solr.core.SolrResourceLoader;
Adding 'file:/opt/solr/solr-4.7.2/solr01/lib/ext/' to classloader
INFO - 2015-08-25 23:40:52.303; org.apache.solr.core.SolrResourceLoader;
Adding
'file:/opt/solr/solr-4.7.2/solr01/lib/jetty-xml-8.1.10.v20130312.jar' to
classloader
so I then ran the split and checked on it in the morning
http://dj01.aws.narasearch.us:8981/solr/admin/collections?action=SPLITSHARD&collection=collection1&shard=shard1
it succeeded and created replicas.
ls /opt/solr/solr-4.7.2/solr0*/solr/
/opt/solr/solr-4.7.2/solr01/solr/:
bin collection1_shard1_0_replica1 README.txt zoo.cfg
collection1 collection1_shard1_1_replica1 solr.xml
/opt/solr/solr-4.7.2/solr02/solr/:
bin collection1 README.txt solr.xml zoo.cfg
/opt/solr/solr-4.7.2/solr03/solr/:
bin collection1 collection1_shard1_0_replica2 README.txt solr.xml
zoo.cfg
/opt/solr/solr-4.7.2/solr04/solr/:
bin collection1 collection1_shard1_1_replica2 README.txt solr.xml
zoo.cfg
and it actually distributed it
[root@dj01 solr]# du -sh *
4.0K bin
41G collection1
18G collection1_shard1_0_replica1
16G collection1_shard1_1_replica1
4.0K README.txt
4.0K solr.xml
4.0K zoo.cfg
[root@dj01 solr]# du -sh
/opt/solr/solr-4.7.2/solr04/solr/collection1_shard1_1_replica2
16G /opt/solr/solr-4.7.2/solr04/solr/collection1_shard1_1_replica2
[root@dj01 solr]# du -sh
/opt/solr/solr-4.7.2/solr03/solr/collection1_shard1_0_replica2
18G /opt/solr/solr-4.7.2/solr03/solr/collection1_shard1_0_replica2
Jeff Courtade
M: 240.507.6116
On Aug 25, 2015 11:09 PM, "Anshum Gupta" <an...@anshumgupta.net> wrote:
> Can you elaborate a bit more on the setup, what do the custom plugins do,
> what error do you get ? It seems like a classloader/classpath issue to me
> which doesn't really relate to Shard splitting.
>
>
> On Tue, Aug 25, 2015 at 7:59 PM, Jeff Courtade <co...@gmail.com>
> wrote:
>
> > I am getting failures when trying too split shards on solr 4.2.7 with
> > custom plugins.
> >
> > It fails regularily it cannot find the jar files for plugins when
> creating
> > the new cores/shards.
> >
> > Ideas?
> >
> > --
> > Thanks,
> >
> > Jeff Courtade
> > M: 240.507.6116
> >
>
>
>
> --
> Anshum Gupta
>
Re: splitting shards on 4.7.2 with custom plugins
Posted by Anshum Gupta <an...@anshumgupta.net>.
Can you elaborate a bit more on the setup, what do the custom plugins do,
what error do you get ? It seems like a classloader/classpath issue to me
which doesn't really relate to Shard splitting.
On Tue, Aug 25, 2015 at 7:59 PM, Jeff Courtade <co...@gmail.com>
wrote:
> I am getting failures when trying too split shards on solr 4.2.7 with
> custom plugins.
>
> It fails regularily it cannot find the jar files for plugins when creating
> the new cores/shards.
>
> Ideas?
>
> --
> Thanks,
>
> Jeff Courtade
> M: 240.507.6116
>
--
Anshum Gupta