You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by davers <db...@improvementdirect.com> on 2013/01/25 19:56:58 UTC
Solr 4.1 Custom Hashing DIH
When I used 4.0 I could use my DIH on any shard and the documents would be
distributed based on the internal hashing algorithm and end up distributed
evenly across my three shards.
I have just upgraded to Solr 4.1 and I have noticed that my documents always
end up on the shard that I run the DIH on. I'm assuming this has something
to do with the changes to allow custom hashing.
How do I use custom hashing? Basically I want to reproduce the functionality
of how Solr 4.0 would hash and distribute the updates automatically but I
don't want to use ID for the hashing function I want to use groupId so that
I can do correct grouping and facet grouping.
Any help or direction would be appreciated,
David
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-1-Custom-Hashing-DIH-tp4036316.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 4.1 Custom Hashing DIH
Posted by davers <db...@improvementdirect.com>.
Also this doesn't seem like a problem with DIH specifically. I tried doing
JSON updates with the same result. Documents are only added to the shard
that I send the updates to.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-1-Custom-Hashing-DIH-tp4036316p4036332.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 4.1 Custom Hashing DIH
Posted by davers <db...@improvementdirect.com>.
I found this: https://issues.apache.org/jira/browse/SOLR-2592
And tried adding the following to my solrconfig.xml
<shardKeyParserFactory
class="org.apache.solr.common.cloud.CompositeIdShardKeyParser">
<str name="clause">groupid</str>
<bool name="clauseRequired">false</bool>
</shardKeyParserFactory>
However I am still getting all documents added to the shard that I run the
DIH on.
According to this comment on the tracker the default sharding should be on
uniqueId anyway
1. Configure a ShardKeyParserFactory in SolrConfig under
config/shardKeyParserFactory. If
there is not one configured the default implementation of sharding on the
document's unique
id will be performed. The default configuration is equivalent to:
{code:xml}
<shardKeyParserFactory class="solr.ShardKeyParserFactory"/>
{code}
So is this a problem with the DIH not respecting sharding? This was not a
problem for me in 4.0 with the same configuration.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-1-Custom-Hashing-DIH-tp4036316p4036321.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 4.1 Custom Hashing DIH
Posted by davers <db...@improvementdirect.com>.
Thank you so much!
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-1-Custom-Hashing-DIH-tp4036316p4036357.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 4.1 Custom Hashing DIH
Posted by Yonik Seeley <yo...@lucidworks.com>.
On Fri, Jan 25, 2013 at 4:09 PM, davers <db...@improvementdirect.com> wrote:
> I'm not sure I understand. I thought ID had to be unique.
Right - the group becomes part of the ID (the prefix), not the whole ID.
> for example
> I have the following
>
> [
> { "id" : 1, "groupid" : 1 },
> { "id" : 2, "groupid" : 1},
> { "id" : 3 "groupid : 2 }
> ]
>
> How would I currently ensure that id 1 & 2 end up on the same shard using
> the ID ?
"id" : "groupX!doc1"
"id" : "groupX!doc2"
"id" : "groupY!doc2"
Or for your data, an abbreviated
"id" : "1!1"
"id" : "1!2"
-Yonik
http://lucidworks.com
Re: Solr 4.1 Custom Hashing DIH
Posted by davers <db...@improvementdirect.com>.
I'm not sure I understand. I thought ID had to be unique.
for example
I have the following
[
{ "id" : 1, "groupid" : 1 },
{ "id" : 2, "groupid" : 1},
{ "id" : 3 "groupid : 2 }
]
How would I currently ensure that id 1 & 2 end up on the same shard using
the ID ?
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-1-Custom-Hashing-DIH-tp4036316p4036354.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 4.1 Custom Hashing DIH
Posted by Yonik Seeley <yo...@lucidworks.com>.
On Fri, Jan 25, 2013 at 3:59 PM, davers <db...@improvementdirect.com> wrote:
> I want to shard on groupid instead of id but it doesn't seem to be working.
That's not yet implemented.
Currently you need to put the group in the ID. From the release notes:
* Simple multi-tenancy through enhanced document routing:
- The "compositeId" router is the default for collections with hash
based routing (i.e. when numShards=N is specified on collection
creation).
- Documents with ids sharing the same domain/prefix, e.g.
'customerB!', will be routed to the same shard, allowing for
efficient querying. At query time, one can specify a "shard.keys"
parameter that lists the domains, e.g. 'shard.keys=customerB!', and
controls what shards the query is routed to.
- Collections that do not specify numShards at collection creation
time use custom sharding and default to the "implicit" router.
Document updates received by a shard will be indexed to that shard,
unless a "_shard_" parameter or document field names a different
shard.
-Yonik
http://lucidworks.com
Re: Solr 4.1 Custom Hashing DIH
Posted by davers <db...@improvementdirect.com>.
Ok that worked thank you!
Now it seems like it is still using the default hashing function.
I put this in my solrconfig.xml under the <config> tag
<shardKeyParserFactory
class="org.apache.solr.common.cloud.CompositeIdShardKeyParser">
<str name="clause">groupid</str>
<bool name="clauseRequired">false</bool>
</shardKeyParserFactory>
I want to shard on groupid instead of id but it doesn't seem to be working.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-1-Custom-Hashing-DIH-tp4036316p4036352.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 4.1 Custom Hashing DIH
Posted by Mark Miller <ma...@gmail.com>.
On Jan 25, 2013, at 3:15 PM, davers <db...@improvementdirect.com> wrote:
> I am using the coreadmin to assign nodes to the cluster. How can I tell the
> collection how many shards it has without using the collections api or
> boostrapping
>
On the first CoreAdmin API call that creates a core, pass the numShards param. The CoreAdmin API works with it.
You can pass it for every call, but the first call is the critical one.
- Mark
Re: Solr 4.1 Custom Hashing DIH
Posted by davers <db...@improvementdirect.com>.
I am using the coreadmin to assign nodes to the cluster. How can I tell the
collection how many shards it has without using the collections api or
boostrapping
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-1-Custom-Hashing-DIH-tp4036316p4036340.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 4.1 Custom Hashing DIH
Posted by Yonik Seeley <yo...@lucidworks.com>.
On Fri, Jan 25, 2013 at 1:56 PM, davers <db...@improvementdirect.com> wrote:
> When I used 4.0 I could use my DIH on any shard and the documents would be
> distributed based on the internal hashing algorithm and end up distributed
> evenly across my three shards.
>
> I have just upgraded to Solr 4.1 and I have noticed that my documents always
> end up on the shard that I run the DIH on. I'm assuming this has something
> to do with the changes to allow custom hashing.
It does... if you create your cluster using numShards=N, then ranges
will be created for each shard and that will be respected when you
index.
If you don't pass numShards, then custom sharding is assumed (i.e. the
user/client decides what document belongs on what shard).
For 4.0, if you didn't pass numShards, then the document was hashed
depending on the current number of shards at the time (but the range
was never recorded anywhere). This was extremely error prone since
adding another shard would then invalidate the mappings of all
previous shards (but no one would know...).
-Yonik
http://lucidworks.com