You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Michael <so...@gmail.com> on 2009/10/07 19:46:54 UTC
Default query parameter for one core
I'd like to have 5 cores on my box. core0 should automatically shard to
cores 1-4, which each have a quarter of my corpus.
I tried this in my solrconfig.xml:
<requestHandler name="standard" class="solr.SearchHandler" default="true">
<lst name="defaults">
<str name="shards">${solr.core.shardsParam:}</str> <!-- aka, if the
core specifies a shardsParam, great, and if not, use nothing -->
</lst>
</requestHandler>
and this in my solr.xml:
<cores adminPath="/admin/cores" shareSchema="true">
<core name="core0" instanceDir="./"
shardsParam="localhost:9990/core1,localhost:9990/core2,localhost:9990/core3,localhost:9990/core4"
/>
<core name="core1" instanceDir="./" dataDir="/home/search/data/1"/>
<!-- etc for cores 2 through 4 -->
</cores>
Unfortunately, this doesn't work, because cores 1 through 4 end up
specifying a blank shards param, which is different from no shards param at
all -- it results in a NullPointerException.
Is there a way to not have the shards param at all for most cores, and for
core0 to specify it?
Re: Default query parameter for one core
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Fri, Oct 9, 2009 at 7:56 PM, Michael <so...@gmail.com> wrote:
> Hm... still no success. Can anyone point me to a doc that explains
> how to define and reference core properties? I've had no luck
> searching Google.
>
> Shalin, I gave an identical '<property name="shardsParam"/>' tag to
> each of my cores, and referenced ${solr.core.shardsParam} (with no
> default specified via a colon) in solrconfig.xml. I get an error on
> startup:
>
>
I should have mentioned it earlier but the property name in your case would
be just ${shardParam}. The "solr.core" string is only for automatically
added properties such as name, instanceDir, dataDir, configName, schemaName
--
Regards,
Shalin Shekhar Mangar.
Re: Default query parameter for one core
Posted by Michael <so...@gmail.com>.
OK, a hacky but working solution to making one core shard to all
others: have the default parameter *name* vary, so that one core gets
"&shards=foo" and all other cores get "&dummy=foo".
# solr.xml
<solr ...>
<property name="shardsKey" value="dummy" />
<property name="shardsValue" value="" />
<cores ...>
<core name="core0" instanceDir="./">
<property name="shardsKey" value="shards" />
<property name="shardsValue" value="localhost:9990/solr/core1,..."/>
</core>
<core name="core1" instanceDir="./" dataDir="/search/1"/>
...
</cores>
</solr>
# solrconfig.xml
<requestHandler ...>
<list name="defaults">
<str name="${shardsKey}">${shardsValue}</str>
...
Michael
On Mon, Oct 12, 2009 at 12:00 PM, Michael <so...@gmail.com> wrote:
> Thanks for your input, Shalin.
>
> On Sun, Oct 11, 2009 at 12:30 AM, Shalin Shekhar Mangar
> <sh...@gmail.com> wrote:
>>> - I can't use a variable like ${shardsParam} in a single shared
>>> solrconfig.xml, because the line
>>> <str name="shards">${shardsParam}</str>
>>> has to be in there, and that forces a (possibly empty) &shards
>>> parameter onto cores that *don't* need one, causing a
>>> NullPointerException.
>>>
>>>
>> Well, we can fix the NPE :) Please raise an issue.
>
> The NPE may be the "correct" behavior -- I'm causing an empty &shards=
> parameter, which doesn't have a defined behavior AFAIK. The
> deficiency I was pointing out was that using ${shardsParam} doesn't
> help me achieve my real goal, which is to have the entire <str> tag
> disappear for some shards.
>
>>> So I think my best bet is to make two mostly-identical
>>> solrconfig.xmls, and point core0 to the one specifying a &shards=
>>> parameter:
>>> <core name="core0" config="core0_solrconfig.xml"/>
>>>
>>> I don't like the duplication of config, but at least it accomplishes my
>>> goal!
>>>
>>>
>> There is another way too. Each plugin in Solr now supports a configuration
>> attribute named "enable" which can be true or false. You can control the
>> value (true/false) through a variable. So you can duplicate just the handle
>> instead of the complete solrconfig.xml
>
> I had looked into this, but thought it doesn't help because I'm not
> disabling an entire plugin -- just a <str> tag specifying a default
> parameter to a <requestHandler>. Individual <str> tags don't have an
> "enable" flag for me to conditionally set to false. Maybe I'm
> misunderstanding what you're suggesting?
>
> Thanks again,
> Michael
>
Re: Default query parameter for one core
Posted by Michael <so...@gmail.com>.
Thanks for your input, Shalin.
On Sun, Oct 11, 2009 at 12:30 AM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
>> - I can't use a variable like ${shardsParam} in a single shared
>> solrconfig.xml, because the line
>> <str name="shards">${shardsParam}</str>
>> has to be in there, and that forces a (possibly empty) &shards
>> parameter onto cores that *don't* need one, causing a
>> NullPointerException.
>>
>>
> Well, we can fix the NPE :) Please raise an issue.
The NPE may be the "correct" behavior -- I'm causing an empty &shards=
parameter, which doesn't have a defined behavior AFAIK. The
deficiency I was pointing out was that using ${shardsParam} doesn't
help me achieve my real goal, which is to have the entire <str> tag
disappear for some shards.
>> So I think my best bet is to make two mostly-identical
>> solrconfig.xmls, and point core0 to the one specifying a &shards=
>> parameter:
>> <core name="core0" config="core0_solrconfig.xml"/>
>>
>> I don't like the duplication of config, but at least it accomplishes my
>> goal!
>>
>>
> There is another way too. Each plugin in Solr now supports a configuration
> attribute named "enable" which can be true or false. You can control the
> value (true/false) through a variable. So you can duplicate just the handle
> instead of the complete solrconfig.xml
I had looked into this, but thought it doesn't help because I'm not
disabling an entire plugin -- just a <str> tag specifying a default
parameter to a <requestHandler>. Individual <str> tags don't have an
"enable" flag for me to conditionally set to false. Maybe I'm
misunderstanding what you're suggesting?
Thanks again,
Michael
Re: Default query parameter for one core
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Fri, Oct 9, 2009 at 9:39 PM, Michael <so...@gmail.com> wrote:
> For posterity...
>
> After reading through http://wiki.apache.org/solr/SolrConfigXml and
> http://wiki.apache.org/solr/CoreAdmin and
> http://issues.apache.org/jira/browse/SOLR-646, I think there's no way
> for me to make only one core specify &shards=foo, short of duplicating
> my solrconfig.xml for that core and adding one line:
>
> - I can't use a variable like ${shardsParam} in a single shared
> solrconfig.xml, because the line
> <str name="shards">${shardsParam}</str>
> has to be in there, and that forces a (possibly empty) &shards
> parameter onto cores that *don't* need one, causing a
> NullPointerException.
>
>
Well, we can fix the NPE :)
Please raise an issue.
> - I can't suck in just that one <str> line via a SOLR-646-style import,
> like
> #solrconfig.xml
> <requestHandler>
> <lst name="defaults">
> <import file="${shardspec_file}"/>
> </list>
> </requestHandler>
> #solr.xml
> <core name="core0"><property name="shardspec_file"
> value="some_file"/>...
> <core name="core1"><property name="shardspec_file"
> value="/dev/null"/>...
> because SOLR-646's <import> feature got cut.
>
> So I think my best bet is to make two mostly-identical
> solrconfig.xmls, and point core0 to the one specifying a &shards=
> parameter:
> <core name="core0" config="core0_solrconfig.xml"/>
>
> I don't like the duplication of config, but at least it accomplishes my
> goal!
>
>
There is another way too. Each plugin in Solr now supports a configuration
attribute named "enable" which can be true or false. You can control the
value (true/false) through a variable. So you can duplicate just the handle
instead of the complete solrconfig.xml
--
Regards,
Shalin Shekhar Mangar.
Re: Default query parameter for one core
Posted by Michael <so...@gmail.com>.
For posterity...
After reading through http://wiki.apache.org/solr/SolrConfigXml and
http://wiki.apache.org/solr/CoreAdmin and
http://issues.apache.org/jira/browse/SOLR-646, I think there's no way
for me to make only one core specify &shards=foo, short of duplicating
my solrconfig.xml for that core and adding one line:
- I can't use a variable like ${shardsParam} in a single shared
solrconfig.xml, because the line
<str name="shards">${shardsParam}</str>
has to be in there, and that forces a (possibly empty) &shards
parameter onto cores that *don't* need one, causing a
NullPointerException.
- I can't suck in just that one <str> line via a SOLR-646-style import, like
#solrconfig.xml
<requestHandler>
<lst name="defaults">
<import file="${shardspec_file}"/>
</list>
</requestHandler>
#solr.xml
<core name="core0"><property name="shardspec_file" value="some_file"/>...
<core name="core1"><property name="shardspec_file" value="/dev/null"/>...
because SOLR-646's <import> feature got cut.
So I think my best bet is to make two mostly-identical
solrconfig.xmls, and point core0 to the one specifying a &shards=
parameter:
<core name="core0" config="core0_solrconfig.xml"/>
I don't like the duplication of config, but at least it accomplishes my goal!
Michael
On Fri, Oct 9, 2009 at 10:37 AM, Michael <so...@gmail.com> wrote:
> On Fri, Oct 9, 2009 at 10:26 AM, Michael <so...@gmail.com> wrote:
>> Hm... still no success. Can anyone point me to a doc that explains
>> how to define and reference core properties? I've had no luck
>> searching Google.
>
> OK, definition is described here:
> http://wiki.apache.org/solr/CoreAdmin#property -- a page I've visited
> many times before but somehow had trouble finding.
>
> Still not sure whether I'm supposed to use "solr.core.shardsParam", or
> just "shardsParam", and how to not have a defaulted &shards parameter
> for most cores.
>
> Michael
>
Re: Default query parameter for one core
Posted by Michael <so...@gmail.com>.
On Fri, Oct 9, 2009 at 10:26 AM, Michael <so...@gmail.com> wrote:
> Hm... still no success. Can anyone point me to a doc that explains
> how to define and reference core properties? I've had no luck
> searching Google.
OK, definition is described here:
http://wiki.apache.org/solr/CoreAdmin#property -- a page I've visited
many times before but somehow had trouble finding.
Still not sure whether I'm supposed to use "solr.core.shardsParam", or
just "shardsParam", and how to not have a defaulted &shards parameter
for most cores.
Michael
Re: Default query parameter for one core
Posted by Michael <so...@gmail.com>.
Hm... still no success. Can anyone point me to a doc that explains
how to define and reference core properties? I've had no luck
searching Google.
Shalin, I gave an identical '<property name="shardsParam"/>' tag to
each of my cores, and referenced ${solr.core.shardsParam} (with no
default specified via a colon) in solrconfig.xml. I get an error on
startup:
SEVERE: Error in solrconfig.xml:org.apache.solr.common.SolrException:
No system property or default value specified for
solr.core.shardsParam
(Not to mention that even if this *did* work, it wouldn't help me with
making most cores *not* specify a &shards query parameter.) Clearly
I'm doing something wrong, but I'm in the dark as to how to do it
right.
Any help would be appreciated!
Michael
On Fri, Oct 9, 2009 at 10:13 AM, Michael <so...@gmail.com> wrote:
> On Fri, Oct 9, 2009 at 6:03 AM, Shalin Shekhar Mangar
> <sh...@gmail.com> wrote:
>> Michael, the last line does not seem right. The <core> tag has nothing
>> called shardParam. If you want to add a core property called shardParam, you
>> need to add something like this:
>>
>> <cores adminPath="/admin/cores" shareSchema="true">
>> <core name="core0" instanceDir="./">
>>
>> <property name="shardsParam"
>> value="localhost:9990/core1,localhost:9990/core2,localhost:9990/core3,localhost:9990/core4"/>
>> </core>
>> </cores>
>
> Thanks, Shalin! I had seen a solrconfig.xml referencing
> ${solr.core.instanceDir} which *is* defined as a <core> attribute, so
> I falsely assumed it would accept arbitrary properties as attributes
> on the <core> tag.
>
Re: Default query parameter for one core
Posted by Michael <so...@gmail.com>.
On Fri, Oct 9, 2009 at 6:03 AM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
> Michael, the last line does not seem right. The <core> tag has nothing
> called shardParam. If you want to add a core property called shardParam, you
> need to add something like this:
>
> <cores adminPath="/admin/cores" shareSchema="true">
> <core name="core0" instanceDir="./">
>
> <property name="shardsParam"
> value="localhost:9990/core1,localhost:9990/core2,localhost:9990/core3,localhost:9990/core4"/>
> </core>
> </cores>
Thanks, Shalin! I had seen a solrconfig.xml referencing
${solr.core.instanceDir} which *is* defined as a <core> attribute, so
I falsely assumed it would accept arbitrary properties as attributes
on the <core> tag.
Re: Default query parameter for one core
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Wed, Oct 7, 2009 at 11:16 PM, Michael <so...@gmail.com> wrote:
> I'd like to have 5 cores on my box. core0 should automatically shard to
> cores 1-4, which each have a quarter of my corpus.
> I tried this in my solrconfig.xml:
>
> <requestHandler name="standard" class="solr.SearchHandler" default="true">
> <lst name="defaults">
> <str name="shards">${solr.core.shardsParam:}</str> <!-- aka, if the
> core specifies a shardsParam, great, and if not, use nothing -->
> </lst>
> </requestHandler>
>
> and this in my solr.xml:
>
> <cores adminPath="/admin/cores" shareSchema="true">
> <core name="core0" instanceDir="./"
>
>
> shardsParam="localhost:9990/core1,localhost:9990/core2,localhost:9990/core3,localhost:9990/core4"
> />
>
Michael, the last line does not seem right. The <core> tag has nothing
called shardParam. If you want to add a core property called shardParam, you
need to add something like this:
<cores adminPath="/admin/cores" shareSchema="true">
<core name="core0" instanceDir="./">
<property name="shardsParam"
value="localhost:9990/core1,localhost:9990/core2,localhost:9990/core3,localhost:9990/core4"/>
</core>
</cores>
--
Regards,
Shalin Shekhar Mangar.
Re: Default query parameter for one core
Posted by Michael <so...@gmail.com>.
On Wed, Oct 7, 2009 at 1:46 PM, Michael <so...@gmail.com> wrote:
> Is there a way to not have the shards param at all for most cores, and for core0 to specify it?
E.g. core0 requests always get a "&shards=foo" appended, while other
cores don't have an "&shards" param at all.
Or, barring that, is there a way to tell one core "use this chunk of
XML for your <defaults> tag", and tell the other cores "use this other
chunk of XML for your <defaults> tag"?