You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by James Strassburg <js...@gmail.com> on 2018/09/05 18:23:56 UTC
Autoscaling with core properties
Hello,
We're creating a SolrCloud in AWS and attempting to use autoscaling to add
replicas during nodeAdded/nodeLost. This was working fine for us until we
started creating collections specifying core properties (e.g.
/solr/admin/collections?action=CREATE&property.synonyms_datasource=REDACTED).
It seems that when the nodeLost/Added trigger fires the properties don't
manifest in the core create invocation and we get errors like the following:
products_20180904200015_shard1_replica_n39:
org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
Could not load conf for core products_20180904200015_shard1_replica_n39:
Can't load schema schema.xml: No system property or default value specified
for synonyms_datasource value:jdbc/${synonyms_datasource}
The autoscaling API also doesn't appear to have a way to set the properties
when we create the triggers.
Are we missing something or is this not supported at this time? I couldn't
find a relevant JIRA or other documentation or solr-user discussion on
this.
thanks,
JiM
Re: Autoscaling with core properties
Posted by James Strassburg <js...@gmail.com>.
I created SOLR-12752 ( https://issues.apache.org/jira/browse/SOLR-12752 )
for this issue. We're also using user properties in our dataimporthandler
data-config.xml so SOLR-11529 ( https://issues.apache.org/jira/browse/SOLR-
11529 <https://issues.apache.org/jira/browse/SOLR-11529> ) prevented us
from validating the Config API (unless we had something configured wrong).
Moving on to SOLR_OPTS to see if that will work but it is undesirable since
the admin UI will display the settings (some of them sensitive).
JiM
On Thu, Sep 6, 2018 at 8:24 AM James Strassburg <js...@gmail.com>
wrote:
> Shalin,
>
> We actually found the ConfigAPI yesterday and started testing that with
> set-user-property. I should know today whether that will work or not and I
> will comment on this thread.
>
> I can open a Jira for the core props and replica props later today as well.
>
> JiM
>
> On Thu, Sep 6, 2018 at 12:37 AM Shalin Shekhar Mangar <
> shalinmangar@gmail.com> wrote:
>
>> Hi Jim,
>>
>> Very interesting scenario that we didn't anticipate. I think this is a
>> limitation of the MoveReplica API which does not move core properties.
>>
>> But it also raises questions such as whether to always move all core
>> properties? I personally think core properties are an artifact that was
>> suitable for non-cloud Solr but does not lend well to cloud environments.
>> We also have replica properties in Solr and corresponding APIs to
>> add/update/delete them. See
>>
>> https://lucene.apache.org/solr/guide/7_4/collections-api.html#addreplicaprop
>> (however, I think even these are not carried forward by move replica
>> today). Do you mind opening a Jira to discuss how we can fix the current
>> behavior?
>>
>> Would using the Config API to set user properties work for your use-case?
>> See
>>
>> https://lucene.apache.org/solr/guide/7_4/configuring-solrconfig-xml.html#substituting-properties-in-solr-config-files
>> and
>>
>> https://lucene.apache.org/solr/guide/7_4/config-api.html#commands-for-user-defined-properties
>>
>> We can improve autoscaling actions such as ComputePlanAction to add custom
>> core properties to any add replica or move replica command. That is
>> probably worth another Jira as well.
>>
>>
>> On Wed, Sep 5, 2018 at 11:54 PM James Strassburg <js...@gmail.com>
>> wrote:
>>
>> > Hello,
>> >
>> > We're creating a SolrCloud in AWS and attempting to use autoscaling to
>> add
>> > replicas during nodeAdded/nodeLost. This was working fine for us until
>> we
>> > started creating collections specifying core properties (e.g.
>> >
>> >
>> /solr/admin/collections?action=CREATE&property.synonyms_datasource=REDACTED).
>> > It seems that when the nodeLost/Added trigger fires the properties don't
>> > manifest in the core create invocation and we get errors like the
>> > following:
>> >
>> > products_20180904200015_shard1_replica_n39:
>> >
>> org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
>> > Could not load conf for core products_20180904200015_shard1_replica_n39:
>> > Can't load schema schema.xml: No system property or default value
>> specified
>> > for synonyms_datasource value:jdbc/${synonyms_datasource}
>> >
>> > The autoscaling API also doesn't appear to have a way to set the
>> properties
>> > when we create the triggers.
>> >
>> > Are we missing something or is this not supported at this time? I
>> couldn't
>> > find a relevant JIRA or other documentation or solr-user discussion on
>> > this.
>> >
>> > thanks,
>> >
>> > JiM
>> >
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
Re: Autoscaling with core properties
Posted by James Strassburg <js...@gmail.com>.
Shalin,
We actually found the ConfigAPI yesterday and started testing that with
set-user-property. I should know today whether that will work or not and I
will comment on this thread.
I can open a Jira for the core props and replica props later today as well.
JiM
On Thu, Sep 6, 2018 at 12:37 AM Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
> Hi Jim,
>
> Very interesting scenario that we didn't anticipate. I think this is a
> limitation of the MoveReplica API which does not move core properties.
>
> But it also raises questions such as whether to always move all core
> properties? I personally think core properties are an artifact that was
> suitable for non-cloud Solr but does not lend well to cloud environments.
> We also have replica properties in Solr and corresponding APIs to
> add/update/delete them. See
>
> https://lucene.apache.org/solr/guide/7_4/collections-api.html#addreplicaprop
> (however, I think even these are not carried forward by move replica
> today). Do you mind opening a Jira to discuss how we can fix the current
> behavior?
>
> Would using the Config API to set user properties work for your use-case?
> See
>
> https://lucene.apache.org/solr/guide/7_4/configuring-solrconfig-xml.html#substituting-properties-in-solr-config-files
> and
>
> https://lucene.apache.org/solr/guide/7_4/config-api.html#commands-for-user-defined-properties
>
> We can improve autoscaling actions such as ComputePlanAction to add custom
> core properties to any add replica or move replica command. That is
> probably worth another Jira as well.
>
>
> On Wed, Sep 5, 2018 at 11:54 PM James Strassburg <js...@gmail.com>
> wrote:
>
> > Hello,
> >
> > We're creating a SolrCloud in AWS and attempting to use autoscaling to
> add
> > replicas during nodeAdded/nodeLost. This was working fine for us until we
> > started creating collections specifying core properties (e.g.
> >
> >
> /solr/admin/collections?action=CREATE&property.synonyms_datasource=REDACTED).
> > It seems that when the nodeLost/Added trigger fires the properties don't
> > manifest in the core create invocation and we get errors like the
> > following:
> >
> > products_20180904200015_shard1_replica_n39:
> >
> org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
> > Could not load conf for core products_20180904200015_shard1_replica_n39:
> > Can't load schema schema.xml: No system property or default value
> specified
> > for synonyms_datasource value:jdbc/${synonyms_datasource}
> >
> > The autoscaling API also doesn't appear to have a way to set the
> properties
> > when we create the triggers.
> >
> > Are we missing something or is this not supported at this time? I
> couldn't
> > find a relevant JIRA or other documentation or solr-user discussion on
> > this.
> >
> > thanks,
> >
> > JiM
> >
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: Autoscaling with core properties
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Hi Jim,
Very interesting scenario that we didn't anticipate. I think this is a
limitation of the MoveReplica API which does not move core properties.
But it also raises questions such as whether to always move all core
properties? I personally think core properties are an artifact that was
suitable for non-cloud Solr but does not lend well to cloud environments.
We also have replica properties in Solr and corresponding APIs to
add/update/delete them. See
https://lucene.apache.org/solr/guide/7_4/collections-api.html#addreplicaprop
(however, I think even these are not carried forward by move replica
today). Do you mind opening a Jira to discuss how we can fix the current
behavior?
Would using the Config API to set user properties work for your use-case?
See
https://lucene.apache.org/solr/guide/7_4/configuring-solrconfig-xml.html#substituting-properties-in-solr-config-files
and
https://lucene.apache.org/solr/guide/7_4/config-api.html#commands-for-user-defined-properties
We can improve autoscaling actions such as ComputePlanAction to add custom
core properties to any add replica or move replica command. That is
probably worth another Jira as well.
On Wed, Sep 5, 2018 at 11:54 PM James Strassburg <js...@gmail.com>
wrote:
> Hello,
>
> We're creating a SolrCloud in AWS and attempting to use autoscaling to add
> replicas during nodeAdded/nodeLost. This was working fine for us until we
> started creating collections specifying core properties (e.g.
>
> /solr/admin/collections?action=CREATE&property.synonyms_datasource=REDACTED).
> It seems that when the nodeLost/Added trigger fires the properties don't
> manifest in the core create invocation and we get errors like the
> following:
>
> products_20180904200015_shard1_replica_n39:
> org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
> Could not load conf for core products_20180904200015_shard1_replica_n39:
> Can't load schema schema.xml: No system property or default value specified
> for synonyms_datasource value:jdbc/${synonyms_datasource}
>
> The autoscaling API also doesn't appear to have a way to set the properties
> when we create the triggers.
>
> Are we missing something or is this not supported at this time? I couldn't
> find a relevant JIRA or other documentation or solr-user discussion on
> this.
>
> thanks,
>
> JiM
>
--
Regards,
Shalin Shekhar Mangar.