Solr Collection Create API queries

[Swathi Singamsetty <sw...@gmail.com>]

Hi Team,

To implement the feature "Persist and use the
replicationFactor,maxShardsPerNode at Collection&Shard level" am following
the steps mentioned in the jira ticket
https://issues.apache.org/jira/browse/SOLR-4808.

I used the "smartCloud" and "autoManageCluster" properties to create a
collection in the create collection API to allow the overseer to bring up
the minimum no. of replicas for each shard as per the replicationFactor set
. But these 2 properties did not persist in the cluster state. Could
someone let me know how to use these properties in this feature?



Thanks & Regards,
Swathi.

Re: Solr Collection Create API queries

[Swathi Singamsetty <sw...@gmail.com>]

Thank you Anshum.
I would try the approach of managing it from outside first and see how it
works.

On Fri, Sep 9, 2016 at 1:51 PM, Anshum Gupta <an...@anshumgupta.net> wrote:

> If you want to build a monitoring tool that maintains a replication factor,
> I would suggest you use the Collections APIs (ClusterStatus, AddReplica,
> DeleteReplica, etc.) and manage this from outside of Solr. I don't want to
> pull you back from trying to build something but I think you'd be biting a
> lot for the first bite if you take this up as the first thing to implement
> within Solr.
>
>
> On Fri, Sep 9, 2016 at 1:41 PM Swathi Singamsetty <
> swathisingamsetty12@gmail.com> wrote:
>
> > I am experimenting on this functionality and see how the overseer
> monitors
> > and keeps the minimum no of replicas up and running.
> >
> >
> > In heavy indexing/search flow , if any replica goes down we need to keep
> > the minimum no. of replicas up and running to serve the traffic and
> > mainitain the availability of the cluster.
> >
> >
> > Please let me know if you need more information.
> >
> > Can you point me to the git repo branch where I can dig deeper and see
> this
> > functionality ?
> >
> >
> >
> > Thanks,
> > Swathi.
> >
> >
> >
> >
> >
> > On Fri, Sep 9, 2016 at 1:10 PM, Anshum Gupta <an...@anshumgupta.net>
> > wrote:
> >
> > > Just to clarify here, I said that I really think it's an XY problem
> > here. I
> > > still don't know what is being attempted/built.
> > >
> > > From the last email, sounds like you want to build/support
> auto-addition
> > of
> > > replica but I would wait until you clarify the use case to suggest
> > > anything.
> > >
> > > -Anshum
> > >
> > > On Fri, Sep 9, 2016 at 8:20 AM Erick Erickson <erickerickson@gmail.com
> >
> > > wrote:
> > >
> > > > I think you're missing my point. The _feature_ may be there,
> > > > you'll have to investigate. But it is not named "smartCloud" or
> > > >  "autoManageCluster". Those terms
> > > > 1> do not appear in the final patch.
> > > > 2> do not appear in any file in Solr 6x.
> > > >
> > > > They were suggested names, what the final implementation
> > > > used should be in the ref guide, although I admit this latter
> > > > sometimes lags.
> > > >
> > > > Best,
> > > > Erick
> > > >
> > > > On Fri, Sep 9, 2016 at 7:51 AM, Swathi Singamsetty
> > > > <sw...@gmail.com> wrote:
> > > > > I am working on solr 6.0.0 to implement this feature.
> > > > > I had a chat with Anshum and confirmed that this feature is
> available
> > > in
> > > > > 6.0.0 version.
> > > > >
> > > > >
> > > > > The functionality is that to allow the overseer to bring up
> > > > >  the minimum no. of replicas for each shard as per the
> > > replicationFactor
> > > > > set.
> > > > >
> > > > > I will look into the ref guide as well.
> > > > >
> > > > > Thanks,
> > > > > Swathi.
> > > > >
> > > > > On Friday, September 9, 2016, Erick Erickson <
> > erickerickson@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> You cannot just pick arbitrary parts of a JIRA discussion
> > > > >> and expect them to work. JIRAs are places where
> > > > >> discussion of alternatives takes place and the discussion
> > > > >> often suggests ideas that are not incorporated
> > > > >> in the final patch. The patch for the JIRA you mentioned,
> > > > >> for instance, does not mention either of those parameters,
> > > > >> which implies that they were simply part of the discussion
> > > > >> and were never implemented.
> > > > >>
> > > > >> So this sounds like an "XY" problem. You're asking why
> > > > >> properties aren't persisted when you really want to take
> > > > >> advantage of some functionality. What is that functionality?
> > > > >>
> > > > >> BTW, I'd go by the ref guide rather than JIRAs unless you
> > > > >> examine the patch and see that the discussion was
> > > > >> implemented in the patch.
> > > > >>
> > > > >> Best,
> > > > >> Erick
> > > > >>
> > > > >> On Thu, Sep 8, 2016 at 9:33 PM, Swathi Singamsetty
> > > > >> <swathisingamsetty12@gmail.com <javascript:;>> wrote:
> > > > >> > Hi Team,
> > > > >> >
> > > > >> > To implement the feature "Persist and use the
> > > > >> > replicationFactor,maxShardsPerNode at Collection&Shard level"
> am
> > > > >> following
> > > > >> > the steps mentioned in the jira ticket
> > > > >> > https://issues.apache.org/jira/browse/SOLR-4808.
> > > > >> >
> > > > >> > I used the "smartCloud" and "autoManageCluster" properties to
> > > create a
> > > > >> > collection in the create collection API to allow the overseer to
> > > > bring up
> > > > >> > the minimum no. of replicas for each shard as per the
> > > > replicationFactor
> > > > >> set
> > > > >> > . But these 2 properties did not persist in the cluster state.
> > Could
> > > > >> > someone let me know how to use these properties in this feature?
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > Thanks & Regards,
> > > > >> > Swathi.
> > > > >>
> > > >
> > >
> >
>

Re: Solr Collection Create API queries

[Anshum Gupta <an...@anshumgupta.net>]

If you want to build a monitoring tool that maintains a replication factor,
I would suggest you use the Collections APIs (ClusterStatus, AddReplica,
DeleteReplica, etc.) and manage this from outside of Solr. I don't want to
pull you back from trying to build something but I think you'd be biting a
lot for the first bite if you take this up as the first thing to implement
within Solr.


On Fri, Sep 9, 2016 at 1:41 PM Swathi Singamsetty <
swathisingamsetty12@gmail.com> wrote:

> I am experimenting on this functionality and see how the overseer monitors
> and keeps the minimum no of replicas up and running.
>
>
> In heavy indexing/search flow , if any replica goes down we need to keep
> the minimum no. of replicas up and running to serve the traffic and
> mainitain the availability of the cluster.
>
>
> Please let me know if you need more information.
>
> Can you point me to the git repo branch where I can dig deeper and see this
> functionality ?
>
>
>
> Thanks,
> Swathi.
>
>
>
>
>
> On Fri, Sep 9, 2016 at 1:10 PM, Anshum Gupta <an...@anshumgupta.net>
> wrote:
>
> > Just to clarify here, I said that I really think it's an XY problem
> here. I
> > still don't know what is being attempted/built.
> >
> > From the last email, sounds like you want to build/support auto-addition
> of
> > replica but I would wait until you clarify the use case to suggest
> > anything.
> >
> > -Anshum
> >
> > On Fri, Sep 9, 2016 at 8:20 AM Erick Erickson <er...@gmail.com>
> > wrote:
> >
> > > I think you're missing my point. The _feature_ may be there,
> > > you'll have to investigate. But it is not named "smartCloud" or
> > >  "autoManageCluster". Those terms
> > > 1> do not appear in the final patch.
> > > 2> do not appear in any file in Solr 6x.
> > >
> > > They were suggested names, what the final implementation
> > > used should be in the ref guide, although I admit this latter
> > > sometimes lags.
> > >
> > > Best,
> > > Erick
> > >
> > > On Fri, Sep 9, 2016 at 7:51 AM, Swathi Singamsetty
> > > <sw...@gmail.com> wrote:
> > > > I am working on solr 6.0.0 to implement this feature.
> > > > I had a chat with Anshum and confirmed that this feature is available
> > in
> > > > 6.0.0 version.
> > > >
> > > >
> > > > The functionality is that to allow the overseer to bring up
> > > >  the minimum no. of replicas for each shard as per the
> > replicationFactor
> > > > set.
> > > >
> > > > I will look into the ref guide as well.
> > > >
> > > > Thanks,
> > > > Swathi.
> > > >
> > > > On Friday, September 9, 2016, Erick Erickson <
> erickerickson@gmail.com>
> > > > wrote:
> > > >
> > > >> You cannot just pick arbitrary parts of a JIRA discussion
> > > >> and expect them to work. JIRAs are places where
> > > >> discussion of alternatives takes place and the discussion
> > > >> often suggests ideas that are not incorporated
> > > >> in the final patch. The patch for the JIRA you mentioned,
> > > >> for instance, does not mention either of those parameters,
> > > >> which implies that they were simply part of the discussion
> > > >> and were never implemented.
> > > >>
> > > >> So this sounds like an "XY" problem. You're asking why
> > > >> properties aren't persisted when you really want to take
> > > >> advantage of some functionality. What is that functionality?
> > > >>
> > > >> BTW, I'd go by the ref guide rather than JIRAs unless you
> > > >> examine the patch and see that the discussion was
> > > >> implemented in the patch.
> > > >>
> > > >> Best,
> > > >> Erick
> > > >>
> > > >> On Thu, Sep 8, 2016 at 9:33 PM, Swathi Singamsetty
> > > >> <swathisingamsetty12@gmail.com <javascript:;>> wrote:
> > > >> > Hi Team,
> > > >> >
> > > >> > To implement the feature "Persist and use the
> > > >> > replicationFactor,maxShardsPerNode at Collection&Shard level" am
> > > >> following
> > > >> > the steps mentioned in the jira ticket
> > > >> > https://issues.apache.org/jira/browse/SOLR-4808.
> > > >> >
> > > >> > I used the "smartCloud" and "autoManageCluster" properties to
> > create a
> > > >> > collection in the create collection API to allow the overseer to
> > > bring up
> > > >> > the minimum no. of replicas for each shard as per the
> > > replicationFactor
> > > >> set
> > > >> > . But these 2 properties did not persist in the cluster state.
> Could
> > > >> > someone let me know how to use these properties in this feature?
> > > >> >
> > > >> >
> > > >> >
> > > >> > Thanks & Regards,
> > > >> > Swathi.
> > > >>
> > >
> >
>

Re: Solr Collection Create API queries

[Swathi Singamsetty <sw...@gmail.com>]

I am experimenting on this functionality and see how the overseer monitors
and keeps the minimum no of replicas up and running.


In heavy indexing/search flow , if any replica goes down we need to keep
the minimum no. of replicas up and running to serve the traffic and
mainitain the availability of the cluster.


Please let me know if you need more information.

Can you point me to the git repo branch where I can dig deeper and see this
functionality ?



Thanks,
Swathi.





On Fri, Sep 9, 2016 at 1:10 PM, Anshum Gupta <an...@anshumgupta.net> wrote:

> Just to clarify here, I said that I really think it's an XY problem here. I
> still don't know what is being attempted/built.
>
> From the last email, sounds like you want to build/support auto-addition of
> replica but I would wait until you clarify the use case to suggest
> anything.
>
> -Anshum
>
> On Fri, Sep 9, 2016 at 8:20 AM Erick Erickson <er...@gmail.com>
> wrote:
>
> > I think you're missing my point. The _feature_ may be there,
> > you'll have to investigate. But it is not named "smartCloud" or
> >  "autoManageCluster". Those terms
> > 1> do not appear in the final patch.
> > 2> do not appear in any file in Solr 6x.
> >
> > They were suggested names, what the final implementation
> > used should be in the ref guide, although I admit this latter
> > sometimes lags.
> >
> > Best,
> > Erick
> >
> > On Fri, Sep 9, 2016 at 7:51 AM, Swathi Singamsetty
> > <sw...@gmail.com> wrote:
> > > I am working on solr 6.0.0 to implement this feature.
> > > I had a chat with Anshum and confirmed that this feature is available
> in
> > > 6.0.0 version.
> > >
> > >
> > > The functionality is that to allow the overseer to bring up
> > >  the minimum no. of replicas for each shard as per the
> replicationFactor
> > > set.
> > >
> > > I will look into the ref guide as well.
> > >
> > > Thanks,
> > > Swathi.
> > >
> > > On Friday, September 9, 2016, Erick Erickson <er...@gmail.com>
> > > wrote:
> > >
> > >> You cannot just pick arbitrary parts of a JIRA discussion
> > >> and expect them to work. JIRAs are places where
> > >> discussion of alternatives takes place and the discussion
> > >> often suggests ideas that are not incorporated
> > >> in the final patch. The patch for the JIRA you mentioned,
> > >> for instance, does not mention either of those parameters,
> > >> which implies that they were simply part of the discussion
> > >> and were never implemented.
> > >>
> > >> So this sounds like an "XY" problem. You're asking why
> > >> properties aren't persisted when you really want to take
> > >> advantage of some functionality. What is that functionality?
> > >>
> > >> BTW, I'd go by the ref guide rather than JIRAs unless you
> > >> examine the patch and see that the discussion was
> > >> implemented in the patch.
> > >>
> > >> Best,
> > >> Erick
> > >>
> > >> On Thu, Sep 8, 2016 at 9:33 PM, Swathi Singamsetty
> > >> <swathisingamsetty12@gmail.com <javascript:;>> wrote:
> > >> > Hi Team,
> > >> >
> > >> > To implement the feature "Persist and use the
> > >> > replicationFactor,maxShardsPerNode at Collection&Shard level" am
> > >> following
> > >> > the steps mentioned in the jira ticket
> > >> > https://issues.apache.org/jira/browse/SOLR-4808.
> > >> >
> > >> > I used the "smartCloud" and "autoManageCluster" properties to
> create a
> > >> > collection in the create collection API to allow the overseer to
> > bring up
> > >> > the minimum no. of replicas for each shard as per the
> > replicationFactor
> > >> set
> > >> > . But these 2 properties did not persist in the cluster state. Could
> > >> > someone let me know how to use these properties in this feature?
> > >> >
> > >> >
> > >> >
> > >> > Thanks & Regards,
> > >> > Swathi.
> > >>
> >
>

Re: Solr Collection Create API queries

[Anshum Gupta <an...@anshumgupta.net>]

Just to clarify here, I said that I really think it's an XY problem here. I
still don't know what is being attempted/built.

From the last email, sounds like you want to build/support auto-addition of
replica but I would wait until you clarify the use case to suggest anything.

-Anshum

On Fri, Sep 9, 2016 at 8:20 AM Erick Erickson <er...@gmail.com>
wrote:

> I think you're missing my point. The _feature_ may be there,
> you'll have to investigate. But it is not named "smartCloud" or
>  "autoManageCluster". Those terms
> 1> do not appear in the final patch.
> 2> do not appear in any file in Solr 6x.
>
> They were suggested names, what the final implementation
> used should be in the ref guide, although I admit this latter
> sometimes lags.
>
> Best,
> Erick
>
> On Fri, Sep 9, 2016 at 7:51 AM, Swathi Singamsetty
> <sw...@gmail.com> wrote:
> > I am working on solr 6.0.0 to implement this feature.
> > I had a chat with Anshum and confirmed that this feature is available in
> > 6.0.0 version.
> >
> >
> > The functionality is that to allow the overseer to bring up
> >  the minimum no. of replicas for each shard as per the replicationFactor
> > set.
> >
> > I will look into the ref guide as well.
> >
> > Thanks,
> > Swathi.
> >
> > On Friday, September 9, 2016, Erick Erickson <er...@gmail.com>
> > wrote:
> >
> >> You cannot just pick arbitrary parts of a JIRA discussion
> >> and expect them to work. JIRAs are places where
> >> discussion of alternatives takes place and the discussion
> >> often suggests ideas that are not incorporated
> >> in the final patch. The patch for the JIRA you mentioned,
> >> for instance, does not mention either of those parameters,
> >> which implies that they were simply part of the discussion
> >> and were never implemented.
> >>
> >> So this sounds like an "XY" problem. You're asking why
> >> properties aren't persisted when you really want to take
> >> advantage of some functionality. What is that functionality?
> >>
> >> BTW, I'd go by the ref guide rather than JIRAs unless you
> >> examine the patch and see that the discussion was
> >> implemented in the patch.
> >>
> >> Best,
> >> Erick
> >>
> >> On Thu, Sep 8, 2016 at 9:33 PM, Swathi Singamsetty
> >> <swathisingamsetty12@gmail.com <javascript:;>> wrote:
> >> > Hi Team,
> >> >
> >> > To implement the feature "Persist and use the
> >> > replicationFactor,maxShardsPerNode at Collection&Shard level" am
> >> following
> >> > the steps mentioned in the jira ticket
> >> > https://issues.apache.org/jira/browse/SOLR-4808.
> >> >
> >> > I used the "smartCloud" and "autoManageCluster" properties to create a
> >> > collection in the create collection API to allow the overseer to
> bring up
> >> > the minimum no. of replicas for each shard as per the
> replicationFactor
> >> set
> >> > . But these 2 properties did not persist in the cluster state. Could
> >> > someone let me know how to use these properties in this feature?
> >> >
> >> >
> >> >
> >> > Thanks & Regards,
> >> > Swathi.
> >>
>

Re: Solr Collection Create API queries

[Erick Erickson <er...@gmail.com>]

I think you're missing my point. The _feature_ may be there,
you'll have to investigate. But it is not named "smartCloud" or
 "autoManageCluster". Those terms
1> do not appear in the final patch.
2> do not appear in any file in Solr 6x.

They were suggested names, what the final implementation
used should be in the ref guide, although I admit this latter
sometimes lags.

Best,
Erick

On Fri, Sep 9, 2016 at 7:51 AM, Swathi Singamsetty
<sw...@gmail.com> wrote:
> I am working on solr 6.0.0 to implement this feature.
> I had a chat with Anshum and confirmed that this feature is available in
> 6.0.0 version.
>
>
> The functionality is that to allow the overseer to bring up
>  the minimum no. of replicas for each shard as per the replicationFactor
> set.
>
> I will look into the ref guide as well.
>
> Thanks,
> Swathi.
>
> On Friday, September 9, 2016, Erick Erickson <er...@gmail.com>
> wrote:
>
>> You cannot just pick arbitrary parts of a JIRA discussion
>> and expect them to work. JIRAs are places where
>> discussion of alternatives takes place and the discussion
>> often suggests ideas that are not incorporated
>> in the final patch. The patch for the JIRA you mentioned,
>> for instance, does not mention either of those parameters,
>> which implies that they were simply part of the discussion
>> and were never implemented.
>>
>> So this sounds like an "XY" problem. You're asking why
>> properties aren't persisted when you really want to take
>> advantage of some functionality. What is that functionality?
>>
>> BTW, I'd go by the ref guide rather than JIRAs unless you
>> examine the patch and see that the discussion was
>> implemented in the patch.
>>
>> Best,
>> Erick
>>
>> On Thu, Sep 8, 2016 at 9:33 PM, Swathi Singamsetty
>> <swathisingamsetty12@gmail.com <javascript:;>> wrote:
>> > Hi Team,
>> >
>> > To implement the feature "Persist and use the
>> > replicationFactor,maxShardsPerNode at Collection&Shard level" am
>> following
>> > the steps mentioned in the jira ticket
>> > https://issues.apache.org/jira/browse/SOLR-4808.
>> >
>> > I used the "smartCloud" and "autoManageCluster" properties to create a
>> > collection in the create collection API to allow the overseer to bring up
>> > the minimum no. of replicas for each shard as per the replicationFactor
>> set
>> > . But these 2 properties did not persist in the cluster state. Could
>> > someone let me know how to use these properties in this feature?
>> >
>> >
>> >
>> > Thanks & Regards,
>> > Swathi.
>>

Re: Solr Collection Create API queries

[Swathi Singamsetty <sw...@gmail.com>]

I am working on solr 6.0.0 to implement this feature.
I had a chat with Anshum and confirmed that this feature is available in
6.0.0 version.


The functionality is that to allow the overseer to bring up
 the minimum no. of replicas for each shard as per the replicationFactor
set.

I will look into the ref guide as well.

Thanks,
Swathi.

On Friday, September 9, 2016, Erick Erickson <er...@gmail.com>
wrote:

> You cannot just pick arbitrary parts of a JIRA discussion
> and expect them to work. JIRAs are places where
> discussion of alternatives takes place and the discussion
> often suggests ideas that are not incorporated
> in the final patch. The patch for the JIRA you mentioned,
> for instance, does not mention either of those parameters,
> which implies that they were simply part of the discussion
> and were never implemented.
>
> So this sounds like an "XY" problem. You're asking why
> properties aren't persisted when you really want to take
> advantage of some functionality. What is that functionality?
>
> BTW, I'd go by the ref guide rather than JIRAs unless you
> examine the patch and see that the discussion was
> implemented in the patch.
>
> Best,
> Erick
>
> On Thu, Sep 8, 2016 at 9:33 PM, Swathi Singamsetty
> <swathisingamsetty12@gmail.com <javascript:;>> wrote:
> > Hi Team,
> >
> > To implement the feature "Persist and use the
> > replicationFactor,maxShardsPerNode at Collection&Shard level" am
> following
> > the steps mentioned in the jira ticket
> > https://issues.apache.org/jira/browse/SOLR-4808.
> >
> > I used the "smartCloud" and "autoManageCluster" properties to create a
> > collection in the create collection API to allow the overseer to bring up
> > the minimum no. of replicas for each shard as per the replicationFactor
> set
> > . But these 2 properties did not persist in the cluster state. Could
> > someone let me know how to use these properties in this feature?
> >
> >
> >
> > Thanks & Regards,
> > Swathi.
>

Re: Solr Collection Create API queries

[Erick Erickson <er...@gmail.com>]

You cannot just pick arbitrary parts of a JIRA discussion
and expect them to work. JIRAs are places where
discussion of alternatives takes place and the discussion
often suggests ideas that are not incorporated
in the final patch. The patch for the JIRA you mentioned,
for instance, does not mention either of those parameters,
which implies that they were simply part of the discussion
and were never implemented.

So this sounds like an "XY" problem. You're asking why
properties aren't persisted when you really want to take
advantage of some functionality. What is that functionality?

BTW, I'd go by the ref guide rather than JIRAs unless you
examine the patch and see that the discussion was
implemented in the patch.

Best,
Erick

On Thu, Sep 8, 2016 at 9:33 PM, Swathi Singamsetty
<sw...@gmail.com> wrote:
> Hi Team,
>
> To implement the feature "Persist and use the
> replicationFactor,maxShardsPerNode at Collection&Shard level" am following
> the steps mentioned in the jira ticket
> https://issues.apache.org/jira/browse/SOLR-4808.
>
> I used the "smartCloud" and "autoManageCluster" properties to create a
> collection in the create collection API to allow the overseer to bring up
> the minimum no. of replicas for each shard as per the replicationFactor set
> . But these 2 properties did not persist in the cluster state. Could
> someone let me know how to use these properties in this feature?
>
>
>
> Thanks & Regards,
> Swathi.