You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Vincenzo D'Amore <v....@gmail.com> on 2022/06/30 15:30:48 UTC
group.ngroups=true and multi shard
Hi,
I have a collection that is becoming bigger than usual (few millions of
documents) so we are doing a few tests to understand how to divide it in
multiple shards (solrcloud 8.11.1).
Strangely since we divided the index in 3 shards the ngroups value is not
always returned.
To be clear, executing the same query 10 times, can happen that 3/4 times
the ngroups value is not returned. We created the shard using
router.name=compositeId
and we have _route_ in the query.
Any suggestions?
Best regards,
Vincenzo
{"device-product-hash":{
"pullReplicas":"0",
"replicationFactor":"3",
"shards":{
"shard1":{
"range":"80000000-d554ffff",
"state":"active",
"replicas":{
"core_node7":{
"core":"device-product-hash_shard1_replica_n4",
"node_name":"solr-1.solrcluster.svc.cluster.local:8983_solr",
"base_url":"
http://solr-1.solrcluster.svc.cluster.local:8983/solr",
"state":"active",
"type":"NRT",
"force_set_state":"false"},
"core_node20":{
"core":"device-product-hash_shard1_replica_n19",
"node_name":"solr-0.solrcluster.svc.cluster.local:8983_solr",
"base_url":"
http://solr-0.solrcluster.svc.cluster.local:8983/solr",
"state":"active",
"type":"NRT",
"force_set_state":"false",
"leader":"true"}}},
"shard2":{
"range":"d5550000-2aa9ffff",
"state":"active",
"replicas":{
"core_node13":{
"core":"device-product-hash_shard2_replica_n10",
"node_name":"solr-1.solrcluster.svc.cluster.local:8983_solr",
"base_url":"
http://solr-1.solrcluster.svc.cluster.local:8983/solr",
"state":"active",
"type":"NRT",
"force_set_state":"false",
"leader":"true"},
"core_node22":{
"core":"device-product-hash_shard2_replica_n21",
"node_name":"solr-2.solrcluster.svc.cluster.local:8983_solr",
"base_url":"
http://solr-2.solrcluster.svc.cluster.local:8983/solr",
"state":"active",
"type":"NRT",
"force_set_state":"false"}}},
"shard3":{
"range":"2aaa0000-7fffffff",
"state":"active",
"replicas":{
"core_node24":{
"core":"device-product-hash_shard3_replica_n23",
"node_name":"solr-0.solrcluster.svc.cluster.local:8983_solr",
"base_url":"
http://solr-0.solrcluster.svc.cluster.local:8983/solr",
"state":"active",
"type":"NRT",
"force_set_state":"false",
"leader":"true"},
"core_node28":{
"core":"device-product-hash_shard3_replica_n27",
"node_name":"solr-2.solrcluster.svc.cluster.local:8983_solr",
"base_url":"
http://solr-2.solrcluster.svc.cluster.local:8983/solr",
"state":"active",
"type":"NRT",
"force_set_state":"false"}}}},
"router":{"name":"compositeId"},
"maxShardsPerNode":"-1",
"autoAddReplicas":"false",
"nrtReplicas":"3",
"tlogReplicas":"0"}}
--
Vincenzo D'Amore
Re: group.ngroups=true and multi shard
Posted by Vincenzo D'Amore <v....@gmail.com>.
Hi Mikhail,
thanks for the prompt reply, not sure I got your suggestion...
Anyway, reading the jira issues I realized that I can have the ngroups
value simply adding a json.facet on the same field I'm grouping to.
Something like this:
json.facet: "{"categories":{"type": "terms","field": "fieldName","limit":
0,"numBuckets":true}}".
Thanks,
Vincenzo
On Thu, Jun 30, 2022 at 8:44 PM Mikhail Khludnev <mk...@apache.org> wrote:
> I've thought JSON Facet is the answer.. despite the question.
> https://issues.apache.org/jira/browse/SOLR-7036
> https://issues.apache.org/jira/browse/SOLR-4763
>
> On Thu, Jun 30, 2022 at 6:31 PM Vincenzo D'Amore <v....@gmail.com>
> wrote:
>
> > Hi,
> >
> > I have a collection that is becoming bigger than usual (few millions of
> > documents) so we are doing a few tests to understand how to divide it in
> > multiple shards (solrcloud 8.11.1).
> >
> > Strangely since we divided the index in 3 shards the ngroups value is not
> > always returned.
> > To be clear, executing the same query 10 times, can happen that 3/4 times
> > the ngroups value is not returned. We created the shard using
> > router.name=compositeId
> > and we have _route_ in the query.
> >
> > Any suggestions?
> >
> > Best regards,
> > Vincenzo
> >
> > {"device-product-hash":{
> > "pullReplicas":"0",
> > "replicationFactor":"3",
> > "shards":{
> > "shard1":{
> > "range":"80000000-d554ffff",
> > "state":"active",
> > "replicas":{
> > "core_node7":{
> > "core":"device-product-hash_shard1_replica_n4",
> > "node_name":"solr-1.solrcluster.svc.cluster.local:8983_solr",
> > "base_url":"
> > http://solr-1.solrcluster.svc.cluster.local:8983/solr",
> > "state":"active",
> > "type":"NRT",
> > "force_set_state":"false"},
> > "core_node20":{
> > "core":"device-product-hash_shard1_replica_n19",
> > "node_name":"solr-0.solrcluster.svc.cluster.local:8983_solr",
> > "base_url":"
> > http://solr-0.solrcluster.svc.cluster.local:8983/solr",
> > "state":"active",
> > "type":"NRT",
> > "force_set_state":"false",
> > "leader":"true"}}},
> > "shard2":{
> > "range":"d5550000-2aa9ffff",
> > "state":"active",
> > "replicas":{
> > "core_node13":{
> > "core":"device-product-hash_shard2_replica_n10",
> > "node_name":"solr-1.solrcluster.svc.cluster.local:8983_solr",
> > "base_url":"
> > http://solr-1.solrcluster.svc.cluster.local:8983/solr",
> > "state":"active",
> > "type":"NRT",
> > "force_set_state":"false",
> > "leader":"true"},
> > "core_node22":{
> > "core":"device-product-hash_shard2_replica_n21",
> > "node_name":"solr-2.solrcluster.svc.cluster.local:8983_solr",
> > "base_url":"
> > http://solr-2.solrcluster.svc.cluster.local:8983/solr",
> > "state":"active",
> > "type":"NRT",
> > "force_set_state":"false"}}},
> > "shard3":{
> > "range":"2aaa0000-7fffffff",
> > "state":"active",
> > "replicas":{
> > "core_node24":{
> > "core":"device-product-hash_shard3_replica_n23",
> > "node_name":"solr-0.solrcluster.svc.cluster.local:8983_solr",
> > "base_url":"
> > http://solr-0.solrcluster.svc.cluster.local:8983/solr",
> > "state":"active",
> > "type":"NRT",
> > "force_set_state":"false",
> > "leader":"true"},
> > "core_node28":{
> > "core":"device-product-hash_shard3_replica_n27",
> > "node_name":"solr-2.solrcluster.svc.cluster.local:8983_solr",
> > "base_url":"
> > http://solr-2.solrcluster.svc.cluster.local:8983/solr",
> > "state":"active",
> > "type":"NRT",
> > "force_set_state":"false"}}}},
> > "router":{"name":"compositeId"},
> > "maxShardsPerNode":"-1",
> > "autoAddReplicas":"false",
> > "nrtReplicas":"3",
> > "tlogReplicas":"0"}}
> >
> >
> >
> >
> > --
> > Vincenzo D'Amore
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
--
Vincenzo D'Amore
Re: group.ngroups=true and multi shard
Posted by Mikhail Khludnev <mk...@apache.org>.
I've thought JSON Facet is the answer.. despite the question.
https://issues.apache.org/jira/browse/SOLR-7036
https://issues.apache.org/jira/browse/SOLR-4763
On Thu, Jun 30, 2022 at 6:31 PM Vincenzo D'Amore <v....@gmail.com> wrote:
> Hi,
>
> I have a collection that is becoming bigger than usual (few millions of
> documents) so we are doing a few tests to understand how to divide it in
> multiple shards (solrcloud 8.11.1).
>
> Strangely since we divided the index in 3 shards the ngroups value is not
> always returned.
> To be clear, executing the same query 10 times, can happen that 3/4 times
> the ngroups value is not returned. We created the shard using
> router.name=compositeId
> and we have _route_ in the query.
>
> Any suggestions?
>
> Best regards,
> Vincenzo
>
> {"device-product-hash":{
> "pullReplicas":"0",
> "replicationFactor":"3",
> "shards":{
> "shard1":{
> "range":"80000000-d554ffff",
> "state":"active",
> "replicas":{
> "core_node7":{
> "core":"device-product-hash_shard1_replica_n4",
> "node_name":"solr-1.solrcluster.svc.cluster.local:8983_solr",
> "base_url":"
> http://solr-1.solrcluster.svc.cluster.local:8983/solr",
> "state":"active",
> "type":"NRT",
> "force_set_state":"false"},
> "core_node20":{
> "core":"device-product-hash_shard1_replica_n19",
> "node_name":"solr-0.solrcluster.svc.cluster.local:8983_solr",
> "base_url":"
> http://solr-0.solrcluster.svc.cluster.local:8983/solr",
> "state":"active",
> "type":"NRT",
> "force_set_state":"false",
> "leader":"true"}}},
> "shard2":{
> "range":"d5550000-2aa9ffff",
> "state":"active",
> "replicas":{
> "core_node13":{
> "core":"device-product-hash_shard2_replica_n10",
> "node_name":"solr-1.solrcluster.svc.cluster.local:8983_solr",
> "base_url":"
> http://solr-1.solrcluster.svc.cluster.local:8983/solr",
> "state":"active",
> "type":"NRT",
> "force_set_state":"false",
> "leader":"true"},
> "core_node22":{
> "core":"device-product-hash_shard2_replica_n21",
> "node_name":"solr-2.solrcluster.svc.cluster.local:8983_solr",
> "base_url":"
> http://solr-2.solrcluster.svc.cluster.local:8983/solr",
> "state":"active",
> "type":"NRT",
> "force_set_state":"false"}}},
> "shard3":{
> "range":"2aaa0000-7fffffff",
> "state":"active",
> "replicas":{
> "core_node24":{
> "core":"device-product-hash_shard3_replica_n23",
> "node_name":"solr-0.solrcluster.svc.cluster.local:8983_solr",
> "base_url":"
> http://solr-0.solrcluster.svc.cluster.local:8983/solr",
> "state":"active",
> "type":"NRT",
> "force_set_state":"false",
> "leader":"true"},
> "core_node28":{
> "core":"device-product-hash_shard3_replica_n27",
> "node_name":"solr-2.solrcluster.svc.cluster.local:8983_solr",
> "base_url":"
> http://solr-2.solrcluster.svc.cluster.local:8983/solr",
> "state":"active",
> "type":"NRT",
> "force_set_state":"false"}}}},
> "router":{"name":"compositeId"},
> "maxShardsPerNode":"-1",
> "autoAddReplicas":"false",
> "nrtReplicas":"3",
> "tlogReplicas":"0"}}
>
>
>
>
> --
> Vincenzo D'Amore
>
--
Sincerely yours
Mikhail Khludnev