You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by David Smiley <ds...@apache.org> on 2024/05/01 18:32:08 UTC
SolrJ collection creation API, replica type specificity
In the interests of supporting different replica types better, I'd
like our SolrJ CollectionAdminRequest methods to not *locally* assume
NRT when creating a request. Calls like createCollection(collection
name, numShards, numReplicas) are nicely ambiguous as to the type, nor
do javadocs indicate what the type is. This is good, I think. Yet
the default behavior is to create a v1 API call that specifies how
many NRT replicas (yes of this specific type) to make. Instead, I'd
like to see the replica type decision made by the server (Solr).
Today it also assumes NRT but I could imagine something as simple as
EnvUtils (env var / sys prop) deciding what the default type should
be. So far this is merely changing CollectionAdminRequest and
consequently the specificity of its v1 requests. It'd be followed up
by improving a number of tests to be less specific to NRT. Any
concerns here?
*Actually* using other replica types (like TLOG or ZERO) may raise
issues for some tests beyond this, sure. In particular, many tests
assume read-your-write (index, commit, query --> find it) but this
won't hold if the query randomly routes to a non-leader. For this I'm
thinking an automatically applied
shards.preference=replica.leader:true
https://solr.apache.org/guide/solr/latest/deployment-guide/solrcloud-distributed-requests.html#shards-preference-parameter
-- only when the default replica type isn't NRT.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: SolrJ collection creation API, replica type specificity
Posted by David Smiley <ds...@apache.org>.
For handlers on a core or collection, defaults are easy to specify in
a <handler> in solrconfig.xml. It'd be cool if we could have a
solr.xml that specifies handlers, and with deeper specificity to the
operations. Kind of a shame to have these dispatching handlers; if it
were at the admin operation level then we'd get cool features for free
like distinct metrics.
On Fri, May 3, 2024 at 3:09 PM David Smiley <ds...@apache.org> wrote:
>
> I totally understand that the client should be empowered to be
> specific, and it is right now. But I also think we should support the
> client being unspecific, and instead allow Solr service owners via
> Solr-side configuration to choose what makes sense. Where I work,
> there are different teams between client and server, the
> people/service at the client don't care about Solr infrastructure
> specifics and new-fangled options (PRS being another) and replica
> types. Updating their client to tweak options around this is
> annoying. They just want a collection to be created, even with an
> assumed configSet as this Solr cluster is only for servicing the needs
> of that client . The Solr service owner (me) is responsible for Solr
> specifics. One could image for one app, assume TLOG and PULL and for
> another, both NRT, or whatever really.
>
> On Fri, May 3, 2024 at 2:58 PM Jason Gerlowski <ge...@gmail.com> wrote:
> >
> > You didn't mention it by name, but it sounds like you're talking about
> > the v1 API's "replicationFactor" parameter (which has defaulted to
> > creating NRT replicas for awhile now)?
> >
> > Personally, I'd rather see that parameter (and corresponding SolrJ
> > code) go away altogether. Some things (e.g. the configset name, the
> > number of shards) are important enough that we force users to be
> > explicit about them...IMO the number and type of replicas fall into
> > that category.
> >
> > But while the ambiguous "replicationFactor" parameter exists I think
> > some sort of "default replica type" concept makes sense. (Granted
> > that we find a way to handle certain complications, like how "PULL"
> > replicas *must* be used in conjunction with replicas of other
> > leader-eligible replica types.)
> >
> > On Wed, May 1, 2024 at 2:32 PM David Smiley <ds...@apache.org> wrote:
> > >
> > > In the interests of supporting different replica types better, I'd
> > > like our SolrJ CollectionAdminRequest methods to not *locally* assume
> > > NRT when creating a request. Calls like createCollection(collection
> > > name, numShards, numReplicas) are nicely ambiguous as to the type, nor
> > > do javadocs indicate what the type is. This is good, I think. Yet
> > > the default behavior is to create a v1 API call that specifies how
> > > many NRT replicas (yes of this specific type) to make. Instead, I'd
> > > like to see the replica type decision made by the server (Solr).
> > > Today it also assumes NRT but I could imagine something as simple as
> > > EnvUtils (env var / sys prop) deciding what the default type should
> > > be. So far this is merely changing CollectionAdminRequest and
> > > consequently the specificity of its v1 requests. It'd be followed up
> > > by improving a number of tests to be less specific to NRT. Any
> > > concerns here?
> > >
> > > *Actually* using other replica types (like TLOG or ZERO) may raise
> > > issues for some tests beyond this, sure. In particular, many tests
> > > assume read-your-write (index, commit, query --> find it) but this
> > > won't hold if the query randomly routes to a non-leader. For this I'm
> > > thinking an automatically applied
> > > shards.preference=replica.leader:true
> > > https://solr.apache.org/guide/solr/latest/deployment-guide/solrcloud-distributed-requests.html#shards-preference-parameter
> > > -- only when the default replica type isn't NRT.
> > >
> > > ~ David Smiley
> > > Apache Lucene/Solr Search Developer
> > > http://www.linkedin.com/in/davidwsmiley
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: SolrJ collection creation API, replica type specificity
Posted by David Smiley <ds...@apache.org>.
I totally understand that the client should be empowered to be
specific, and it is right now. But I also think we should support the
client being unspecific, and instead allow Solr service owners via
Solr-side configuration to choose what makes sense. Where I work,
there are different teams between client and server, the
people/service at the client don't care about Solr infrastructure
specifics and new-fangled options (PRS being another) and replica
types. Updating their client to tweak options around this is
annoying. They just want a collection to be created, even with an
assumed configSet as this Solr cluster is only for servicing the needs
of that client . The Solr service owner (me) is responsible for Solr
specifics. One could image for one app, assume TLOG and PULL and for
another, both NRT, or whatever really.
On Fri, May 3, 2024 at 2:58 PM Jason Gerlowski <ge...@gmail.com> wrote:
>
> You didn't mention it by name, but it sounds like you're talking about
> the v1 API's "replicationFactor" parameter (which has defaulted to
> creating NRT replicas for awhile now)?
>
> Personally, I'd rather see that parameter (and corresponding SolrJ
> code) go away altogether. Some things (e.g. the configset name, the
> number of shards) are important enough that we force users to be
> explicit about them...IMO the number and type of replicas fall into
> that category.
>
> But while the ambiguous "replicationFactor" parameter exists I think
> some sort of "default replica type" concept makes sense. (Granted
> that we find a way to handle certain complications, like how "PULL"
> replicas *must* be used in conjunction with replicas of other
> leader-eligible replica types.)
>
> On Wed, May 1, 2024 at 2:32 PM David Smiley <ds...@apache.org> wrote:
> >
> > In the interests of supporting different replica types better, I'd
> > like our SolrJ CollectionAdminRequest methods to not *locally* assume
> > NRT when creating a request. Calls like createCollection(collection
> > name, numShards, numReplicas) are nicely ambiguous as to the type, nor
> > do javadocs indicate what the type is. This is good, I think. Yet
> > the default behavior is to create a v1 API call that specifies how
> > many NRT replicas (yes of this specific type) to make. Instead, I'd
> > like to see the replica type decision made by the server (Solr).
> > Today it also assumes NRT but I could imagine something as simple as
> > EnvUtils (env var / sys prop) deciding what the default type should
> > be. So far this is merely changing CollectionAdminRequest and
> > consequently the specificity of its v1 requests. It'd be followed up
> > by improving a number of tests to be less specific to NRT. Any
> > concerns here?
> >
> > *Actually* using other replica types (like TLOG or ZERO) may raise
> > issues for some tests beyond this, sure. In particular, many tests
> > assume read-your-write (index, commit, query --> find it) but this
> > won't hold if the query randomly routes to a non-leader. For this I'm
> > thinking an automatically applied
> > shards.preference=replica.leader:true
> > https://solr.apache.org/guide/solr/latest/deployment-guide/solrcloud-distributed-requests.html#shards-preference-parameter
> > -- only when the default replica type isn't NRT.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: SolrJ collection creation API, replica type specificity
Posted by Jason Gerlowski <ge...@gmail.com>.
You didn't mention it by name, but it sounds like you're talking about
the v1 API's "replicationFactor" parameter (which has defaulted to
creating NRT replicas for awhile now)?
Personally, I'd rather see that parameter (and corresponding SolrJ
code) go away altogether. Some things (e.g. the configset name, the
number of shards) are important enough that we force users to be
explicit about them...IMO the number and type of replicas fall into
that category.
But while the ambiguous "replicationFactor" parameter exists I think
some sort of "default replica type" concept makes sense. (Granted
that we find a way to handle certain complications, like how "PULL"
replicas *must* be used in conjunction with replicas of other
leader-eligible replica types.)
On Wed, May 1, 2024 at 2:32 PM David Smiley <ds...@apache.org> wrote:
>
> In the interests of supporting different replica types better, I'd
> like our SolrJ CollectionAdminRequest methods to not *locally* assume
> NRT when creating a request. Calls like createCollection(collection
> name, numShards, numReplicas) are nicely ambiguous as to the type, nor
> do javadocs indicate what the type is. This is good, I think. Yet
> the default behavior is to create a v1 API call that specifies how
> many NRT replicas (yes of this specific type) to make. Instead, I'd
> like to see the replica type decision made by the server (Solr).
> Today it also assumes NRT but I could imagine something as simple as
> EnvUtils (env var / sys prop) deciding what the default type should
> be. So far this is merely changing CollectionAdminRequest and
> consequently the specificity of its v1 requests. It'd be followed up
> by improving a number of tests to be less specific to NRT. Any
> concerns here?
>
> *Actually* using other replica types (like TLOG or ZERO) may raise
> issues for some tests beyond this, sure. In particular, many tests
> assume read-your-write (index, commit, query --> find it) but this
> won't hold if the query randomly routes to a non-leader. For this I'm
> thinking an automatically applied
> shards.preference=replica.leader:true
> https://solr.apache.org/guide/solr/latest/deployment-guide/solrcloud-distributed-requests.html#shards-preference-parameter
> -- only when the default replica type isn't NRT.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org