You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Lorenzo Fundaró <lo...@dawandamail.com> on 2016/07/04 13:46:49 UTC
stateless solr ?
Hello guys,
I am trying to run Solr on my infrastructure using docker containers and
Mesos. My problem is that I don't have a shared filesystem. I have a
cluster of 3 shards and 3 replicas (9 nodes in total) so if I distribute
well my nodes I always have 2 fallbacks of my data for every shard. Every
solr node will store the index in its internal docker filesystem. My
problem is that if I want to relocate a certain node (maybe an automatic
relocation because of a hardware failure), I need to create the core
manually in the new node because it's expecting to find the core.properties
file in the data folder and of course it won't because the storage is
ephemeral. Is there a way to make a new node join the cluster with no
manual intervention ?
Thanks in advance !
--
--
Lorenzo Fundaro
Backend Engineer
E-Mail: lorenzo.fundaro@dawandamail.com
Fax + 49 - (0)30 - 25 76 08 52
Tel + 49 - (0)179 - 51 10 982
DaWanda GmbH
Windscheidstraße 18
10627 Berlin
Geschäftsführer: Claudia Helming, Niels Nüssler und Michael Pütz
AG Charlottenburg HRB 104695 B http://www.dawanda.com
Re: stateless solr ?
Posted by Lorenzo Fundaró <lo...@dawandamail.com>.
Hi guys. Thank you for your answers. @Upayavira I'm interested on your
presentation. @steven im also interested on this patch you're working on.
I actuallly believe that a solution is can be something like a supervisor
or sidekick that knows when the containers die and provisions the new ones
with the right core configuration. This will have to make the corresponding
api calls to solr of course.
Any github project where i can take a look to the Progress you are making?
Thanks
On Jul 5, 2016 01:27, "Steven Bower" <sb...@alcyon.net> wrote:
I don't think that's a bad approach with the sidecar.. We run a huge number
of solr ~5k instances so adding sidecars for each one ads a lot of extra
containers..
What I mean by transition is a container dying and a new one being brought
online to replace it.. With the mod we are working on you won't need the
sidecar to add cores to the new node via the API and remove the old cores..
A new instance would start up with the same node name and just take over
the existing cores (of course will require replication but that will happen
automatically )
Steve
On Mon, Jul 4, 2016 at 5:27 PM Upayavira <uv...@odoko.co.uk> wrote:
> What do you mean by a "transition"?
>
> Can you configure a sidekick container within your orchestrator? Have a
> sidekick always run alongside your SolrCloud nodes? In which case, this
> would be an app that does the calling of the API for you.
>
> Upayavira
>
> On Mon, 4 Jul 2016, at 08:53 PM, Steven Bower wrote:
> > My main issue is having to make any solr collection api calls during a
> > transition.. It makes integrating with orchestration engines way more
> > complex..
> > On Mon, Jul 4, 2016 at 3:40 PM Upayavira <uv...@odoko.co.uk> wrote:
> >
> > > Are you using Solrcloud? With Solrcloud this stuff is easy. You just
> add
> > > a new replica for a collection, and the data is added to the new host.
> > >
> > > I'm working on a demo that will show this all working within Docker
and
> > > Rancher. I've got some code (which I will open source) that handles
> > > config uploads, collection creation, etc. You can add a replica by
> > > running a container on the same node as you want the replica to
reside,
> > > it'll do the rest for you.
> > >
> > > I've got the Solr bit more or less done, I'm now working on everything
> > > else (Dockerised Docker Registry/Jenkins, AWS infra build, etc).
> > >
> > > Let me know if this is interesting to you. If so, I'll post it here
> when
> > > I'm done with it.
> > >
> > > Upayavira
> > >
> > > On Mon, 4 Jul 2016, at 02:46 PM, Lorenzo Fundaró wrote:
> > > > Hello guys,
> > > >
> > > > I am trying to run Solr on my infrastructure using docker containers
> and
> > > > Mesos. My problem is that I don't have a shared filesystem. I have a
> > > > cluster of 3 shards and 3 replicas (9 nodes in total) so if I
> distribute
> > > > well my nodes I always have 2 fallbacks of my data for every shard.
> Every
> > > > solr node will store the index in its internal docker filesystem. My
> > > > problem is that if I want to relocate a certain node (maybe an
> automatic
> > > > relocation because of a hardware failure), I need to create the core
> > > > manually in the new node because it's expecting to find the
> > > > core.properties
> > > > file in the data folder and of course it won't because the storage
is
> > > > ephemeral. Is there a way to make a new node join the cluster with
no
> > > > manual intervention ?
> > > >
> > > > Thanks in advance !
> > > >
> > > >
> > > > --
> > > >
> > > > --
> > > > Lorenzo Fundaro
> > > > Backend Engineer
> > > > E-Mail: lorenzo.fundaro@dawandamail.com
> > > >
> > > > Fax + 49 - (0)30 - 25 76 08 52
> > > > Tel + 49 - (0)179 - 51 10 982
> > > >
> > > > DaWanda GmbH
> > > > Windscheidstraße 18
> > > > 10627 Berlin
> > > >
> > > > Geschäftsführer: Claudia Helming, Niels Nüssler und Michael Pütz
> > > > AG Charlottenburg HRB 104695 B http://www.dawanda.com
> > >
>
Re: stateless solr ?
Posted by Steven Bower <sb...@alcyon.net>.
I don't think that's a bad approach with the sidecar.. We run a huge number
of solr ~5k instances so adding sidecars for each one ads a lot of extra
containers..
What I mean by transition is a container dying and a new one being brought
online to replace it.. With the mod we are working on you won't need the
sidecar to add cores to the new node via the API and remove the old cores..
A new instance would start up with the same node name and just take over
the existing cores (of course will require replication but that will happen
automatically )
Steve
On Mon, Jul 4, 2016 at 5:27 PM Upayavira <uv...@odoko.co.uk> wrote:
> What do you mean by a "transition"?
>
> Can you configure a sidekick container within your orchestrator? Have a
> sidekick always run alongside your SolrCloud nodes? In which case, this
> would be an app that does the calling of the API for you.
>
> Upayavira
>
> On Mon, 4 Jul 2016, at 08:53 PM, Steven Bower wrote:
> > My main issue is having to make any solr collection api calls during a
> > transition.. It makes integrating with orchestration engines way more
> > complex..
> > On Mon, Jul 4, 2016 at 3:40 PM Upayavira <uv...@odoko.co.uk> wrote:
> >
> > > Are you using Solrcloud? With Solrcloud this stuff is easy. You just
> add
> > > a new replica for a collection, and the data is added to the new host.
> > >
> > > I'm working on a demo that will show this all working within Docker and
> > > Rancher. I've got some code (which I will open source) that handles
> > > config uploads, collection creation, etc. You can add a replica by
> > > running a container on the same node as you want the replica to reside,
> > > it'll do the rest for you.
> > >
> > > I've got the Solr bit more or less done, I'm now working on everything
> > > else (Dockerised Docker Registry/Jenkins, AWS infra build, etc).
> > >
> > > Let me know if this is interesting to you. If so, I'll post it here
> when
> > > I'm done with it.
> > >
> > > Upayavira
> > >
> > > On Mon, 4 Jul 2016, at 02:46 PM, Lorenzo Fundaró wrote:
> > > > Hello guys,
> > > >
> > > > I am trying to run Solr on my infrastructure using docker containers
> and
> > > > Mesos. My problem is that I don't have a shared filesystem. I have a
> > > > cluster of 3 shards and 3 replicas (9 nodes in total) so if I
> distribute
> > > > well my nodes I always have 2 fallbacks of my data for every shard.
> Every
> > > > solr node will store the index in its internal docker filesystem. My
> > > > problem is that if I want to relocate a certain node (maybe an
> automatic
> > > > relocation because of a hardware failure), I need to create the core
> > > > manually in the new node because it's expecting to find the
> > > > core.properties
> > > > file in the data folder and of course it won't because the storage is
> > > > ephemeral. Is there a way to make a new node join the cluster with no
> > > > manual intervention ?
> > > >
> > > > Thanks in advance !
> > > >
> > > >
> > > > --
> > > >
> > > > --
> > > > Lorenzo Fundaro
> > > > Backend Engineer
> > > > E-Mail: lorenzo.fundaro@dawandamail.com
> > > >
> > > > Fax + 49 - (0)30 - 25 76 08 52
> > > > Tel + 49 - (0)179 - 51 10 982
> > > >
> > > > DaWanda GmbH
> > > > Windscheidstraße 18
> > > > 10627 Berlin
> > > >
> > > > Geschäftsführer: Claudia Helming, Niels Nüssler und Michael Pütz
> > > > AG Charlottenburg HRB 104695 B http://www.dawanda.com
> > >
>
Re: stateless solr ?
Posted by Upayavira <uv...@odoko.co.uk>.
What do you mean by a "transition"?
Can you configure a sidekick container within your orchestrator? Have a
sidekick always run alongside your SolrCloud nodes? In which case, this
would be an app that does the calling of the API for you.
Upayavira
On Mon, 4 Jul 2016, at 08:53 PM, Steven Bower wrote:
> My main issue is having to make any solr collection api calls during a
> transition.. It makes integrating with orchestration engines way more
> complex..
> On Mon, Jul 4, 2016 at 3:40 PM Upayavira <uv...@odoko.co.uk> wrote:
>
> > Are you using Solrcloud? With Solrcloud this stuff is easy. You just add
> > a new replica for a collection, and the data is added to the new host.
> >
> > I'm working on a demo that will show this all working within Docker and
> > Rancher. I've got some code (which I will open source) that handles
> > config uploads, collection creation, etc. You can add a replica by
> > running a container on the same node as you want the replica to reside,
> > it'll do the rest for you.
> >
> > I've got the Solr bit more or less done, I'm now working on everything
> > else (Dockerised Docker Registry/Jenkins, AWS infra build, etc).
> >
> > Let me know if this is interesting to you. If so, I'll post it here when
> > I'm done with it.
> >
> > Upayavira
> >
> > On Mon, 4 Jul 2016, at 02:46 PM, Lorenzo Fundaró wrote:
> > > Hello guys,
> > >
> > > I am trying to run Solr on my infrastructure using docker containers and
> > > Mesos. My problem is that I don't have a shared filesystem. I have a
> > > cluster of 3 shards and 3 replicas (9 nodes in total) so if I distribute
> > > well my nodes I always have 2 fallbacks of my data for every shard. Every
> > > solr node will store the index in its internal docker filesystem. My
> > > problem is that if I want to relocate a certain node (maybe an automatic
> > > relocation because of a hardware failure), I need to create the core
> > > manually in the new node because it's expecting to find the
> > > core.properties
> > > file in the data folder and of course it won't because the storage is
> > > ephemeral. Is there a way to make a new node join the cluster with no
> > > manual intervention ?
> > >
> > > Thanks in advance !
> > >
> > >
> > > --
> > >
> > > --
> > > Lorenzo Fundaro
> > > Backend Engineer
> > > E-Mail: lorenzo.fundaro@dawandamail.com
> > >
> > > Fax + 49 - (0)30 - 25 76 08 52
> > > Tel + 49 - (0)179 - 51 10 982
> > >
> > > DaWanda GmbH
> > > Windscheidstraße 18
> > > 10627 Berlin
> > >
> > > Geschäftsführer: Claudia Helming, Niels Nüssler und Michael Pütz
> > > AG Charlottenburg HRB 104695 B http://www.dawanda.com
> >
Re: stateless solr ?
Posted by Steven Bower <sb...@alcyon.net>.
My main issue is having to make any solr collection api calls during a
transition.. It makes integrating with orchestration engines way more
complex..
On Mon, Jul 4, 2016 at 3:40 PM Upayavira <uv...@odoko.co.uk> wrote:
> Are you using Solrcloud? With Solrcloud this stuff is easy. You just add
> a new replica for a collection, and the data is added to the new host.
>
> I'm working on a demo that will show this all working within Docker and
> Rancher. I've got some code (which I will open source) that handles
> config uploads, collection creation, etc. You can add a replica by
> running a container on the same node as you want the replica to reside,
> it'll do the rest for you.
>
> I've got the Solr bit more or less done, I'm now working on everything
> else (Dockerised Docker Registry/Jenkins, AWS infra build, etc).
>
> Let me know if this is interesting to you. If so, I'll post it here when
> I'm done with it.
>
> Upayavira
>
> On Mon, 4 Jul 2016, at 02:46 PM, Lorenzo Fundaró wrote:
> > Hello guys,
> >
> > I am trying to run Solr on my infrastructure using docker containers and
> > Mesos. My problem is that I don't have a shared filesystem. I have a
> > cluster of 3 shards and 3 replicas (9 nodes in total) so if I distribute
> > well my nodes I always have 2 fallbacks of my data for every shard. Every
> > solr node will store the index in its internal docker filesystem. My
> > problem is that if I want to relocate a certain node (maybe an automatic
> > relocation because of a hardware failure), I need to create the core
> > manually in the new node because it's expecting to find the
> > core.properties
> > file in the data folder and of course it won't because the storage is
> > ephemeral. Is there a way to make a new node join the cluster with no
> > manual intervention ?
> >
> > Thanks in advance !
> >
> >
> > --
> >
> > --
> > Lorenzo Fundaro
> > Backend Engineer
> > E-Mail: lorenzo.fundaro@dawandamail.com
> >
> > Fax + 49 - (0)30 - 25 76 08 52
> > Tel + 49 - (0)179 - 51 10 982
> >
> > DaWanda GmbH
> > Windscheidstraße 18
> > 10627 Berlin
> >
> > Geschäftsführer: Claudia Helming, Niels Nüssler und Michael Pütz
> > AG Charlottenburg HRB 104695 B http://www.dawanda.com
>
Re: stateless solr ?
Posted by Upayavira <uv...@odoko.co.uk>.
Are you using Solrcloud? With Solrcloud this stuff is easy. You just add
a new replica for a collection, and the data is added to the new host.
I'm working on a demo that will show this all working within Docker and
Rancher. I've got some code (which I will open source) that handles
config uploads, collection creation, etc. You can add a replica by
running a container on the same node as you want the replica to reside,
it'll do the rest for you.
I've got the Solr bit more or less done, I'm now working on everything
else (Dockerised Docker Registry/Jenkins, AWS infra build, etc).
Let me know if this is interesting to you. If so, I'll post it here when
I'm done with it.
Upayavira
On Mon, 4 Jul 2016, at 02:46 PM, Lorenzo Fundaró wrote:
> Hello guys,
>
> I am trying to run Solr on my infrastructure using docker containers and
> Mesos. My problem is that I don't have a shared filesystem. I have a
> cluster of 3 shards and 3 replicas (9 nodes in total) so if I distribute
> well my nodes I always have 2 fallbacks of my data for every shard. Every
> solr node will store the index in its internal docker filesystem. My
> problem is that if I want to relocate a certain node (maybe an automatic
> relocation because of a hardware failure), I need to create the core
> manually in the new node because it's expecting to find the
> core.properties
> file in the data folder and of course it won't because the storage is
> ephemeral. Is there a way to make a new node join the cluster with no
> manual intervention ?
>
> Thanks in advance !
>
>
> --
>
> --
> Lorenzo Fundaro
> Backend Engineer
> E-Mail: lorenzo.fundaro@dawandamail.com
>
> Fax + 49 - (0)30 - 25 76 08 52
> Tel + 49 - (0)179 - 51 10 982
>
> DaWanda GmbH
> Windscheidstraße 18
> 10627 Berlin
>
> Geschäftsführer: Claudia Helming, Niels Nüssler und Michael Pütz
> AG Charlottenburg HRB 104695 B http://www.dawanda.com
Re: stateless solr ?
Posted by Steven Bower <sb...@alcyon.net>.
We have been working on some changes that should help with this.. 1st
challenge is having the node name remain static regardless of where the
node runs (right now it uses host and port, so this won't work unless you
are using some sort of tunneled or dynamic networking).. We have a patch we
are working on for this.. Once this is in place and you use the "zookeeper
is truth" mode for solr cloud and this should seamlessly transition into
the new node (and replicate).. Will update with the ticket number as I
forget it off hand
Steve
On Mon, Jul 4, 2016 at 9:47 AM Lorenzo Fundaró <
lorenzo.fundaro@dawandamail.com> wrote:
> Hello guys,
>
> I am trying to run Solr on my infrastructure using docker containers and
> Mesos. My problem is that I don't have a shared filesystem. I have a
> cluster of 3 shards and 3 replicas (9 nodes in total) so if I distribute
> well my nodes I always have 2 fallbacks of my data for every shard. Every
> solr node will store the index in its internal docker filesystem. My
> problem is that if I want to relocate a certain node (maybe an automatic
> relocation because of a hardware failure), I need to create the core
> manually in the new node because it's expecting to find the core.properties
> file in the data folder and of course it won't because the storage is
> ephemeral. Is there a way to make a new node join the cluster with no
> manual intervention ?
>
> Thanks in advance !
>
>
> --
>
> --
> Lorenzo Fundaro
> Backend Engineer
> E-Mail: lorenzo.fundaro@dawandamail.com
>
> Fax + 49 - (0)30 - 25 76 08 52
> Tel + 49 - (0)179 - 51 10 982
>
> DaWanda GmbH
> Windscheidstraße 18
> 10627 Berlin
>
> Geschäftsführer: Claudia Helming, Niels Nüssler und Michael Pütz
> AG Charlottenburg HRB 104695 B http://www.dawanda.com
>
Re: stateless solr ?
Posted by Steven Bower <sb...@alcyon.net>.
The ticket in question is https://issues.apache.org/jira/browse/SOLR-9265
We are working on a patch now... will update when we have a working patch /
tests..
Shawn is correct that when adding a new node to a SolrCloud cluster it will
not automatically add replicas/etc..
The idea behind this patch though is that you'll be hard-coding the
node-names (eg node1, node2, etc..) that are are normally generated from
the host/port of the instance and as such not actually adding a "new" node
but replacing an existing node. In this case Solr will see the existing
cores and replicate the data down.. At least in theory.. We will need to
verify this assumption with a lot of testing, especially around having
multiple nodes with the same node name showing up at the same time (which
is something currently that cannot really occur, but can once we do this
work)..
steve
On Tue, Jul 5, 2016 at 9:54 AM Shawn Heisey <ap...@elyograg.org> wrote:
> On 7/4/2016 7:46 AM, Lorenzo Fundaró wrote:
> > I am trying to run Solr on my infrastructure using docker containers
> > and Mesos. My problem is that I don't have a shared filesystem. I have
> > a cluster of 3 shards and 3 replicas (9 nodes in total) so if I
> > distribute well my nodes I always have 2 fallbacks of my data for
> > every shard. Every solr node will store the index in its internal
> > docker filesystem. My problem is that if I want to relocate a certain
> > node (maybe an automatic relocation because of a hardware failure), I
> > need to create the core manually in the new node because it's
> > expecting to find the core.properties file in the data folder and of
> > course it won't because the storage is ephemeral. Is there a way to
> > make a new node join the cluster with no manual intervention ?
>
> The things you're asking sound like SolrCloud. The rest of this message
> assumes that you're running cloud. If you're not, then we may need to
> start over.
>
> When you start a new node, it automatically joins the cluster described
> by the Zookeeper database that you point it to.
>
> SolrCloud will **NOT** automatically create replicas when a new node
> joins the cluster. There's no way for SolrCloud to know what you
> actually want to use that new node for, so anything that it did
> automatically might be completely the wrong thing.
>
> Once you add a new node, you can replicate existing data to it with the
> ADDREPLICA action on the Collections API:
>
>
> https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api_addreplica
>
> If the original problem was a down node, you might also want to use the
> DELETEREPLICA action to delete any replicas on the node that you lost
> that are marked down:
>
>
> https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api9
>
> Creating cores manually in your situation is not advisable. The
> CoreAdmin API should not be used when you're running in cloud mode.
>
> Thanks,
> Shawn
>
>
Re: stateless solr ?
Posted by Lorenzo Fundaró <lo...@dawandamail.com>.
Yes,
as a matter of fact I ended up doing all those steps you described. When a
new node joins I check if a shard is missing a replica, if it does then I
add a live node not being used by other shards using the ADDREPLICA call.
Same thing applies for when I found down nodes, I just delete them using
the DELETEREPLICA call.
thank you for the input ! :)
On 5 July 2016 at 15:54, Shawn Heisey <ap...@elyograg.org> wrote:
> On 7/4/2016 7:46 AM, Lorenzo Fundaró wrote:
> > I am trying to run Solr on my infrastructure using docker containers
> > and Mesos. My problem is that I don't have a shared filesystem. I have
> > a cluster of 3 shards and 3 replicas (9 nodes in total) so if I
> > distribute well my nodes I always have 2 fallbacks of my data for
> > every shard. Every solr node will store the index in its internal
> > docker filesystem. My problem is that if I want to relocate a certain
> > node (maybe an automatic relocation because of a hardware failure), I
> > need to create the core manually in the new node because it's
> > expecting to find the core.properties file in the data folder and of
> > course it won't because the storage is ephemeral. Is there a way to
> > make a new node join the cluster with no manual intervention ?
>
> The things you're asking sound like SolrCloud. The rest of this message
> assumes that you're running cloud. If you're not, then we may need to
> start over.
>
> When you start a new node, it automatically joins the cluster described
> by the Zookeeper database that you point it to.
>
> SolrCloud will **NOT** automatically create replicas when a new node
> joins the cluster. There's no way for SolrCloud to know what you
> actually want to use that new node for, so anything that it did
> automatically might be completely the wrong thing.
>
> Once you add a new node, you can replicate existing data to it with the
> ADDREPLICA action on the Collections API:
>
>
> https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api_addreplica
>
> If the original problem was a down node, you might also want to use the
> DELETEREPLICA action to delete any replicas on the node that you lost
> that are marked down:
>
>
> https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api9
>
> Creating cores manually in your situation is not advisable. The
> CoreAdmin API should not be used when you're running in cloud mode.
>
> Thanks,
> Shawn
>
>
--
--
Lorenzo Fundaro
Backend Engineer
E-Mail: lorenzo.fundaro@dawandamail.com
Fax + 49 - (0)30 - 25 76 08 52
Tel + 49 - (0)179 - 51 10 982
DaWanda GmbH
Windscheidstraße 18
10627 Berlin
Geschäftsführer: Claudia Helming und Niels Nüssler
AG Charlottenburg HRB 104695 B http://www.dawanda.com
Re: stateless solr ?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/4/2016 7:46 AM, Lorenzo Fundar� wrote:
> I am trying to run Solr on my infrastructure using docker containers
> and Mesos. My problem is that I don't have a shared filesystem. I have
> a cluster of 3 shards and 3 replicas (9 nodes in total) so if I
> distribute well my nodes I always have 2 fallbacks of my data for
> every shard. Every solr node will store the index in its internal
> docker filesystem. My problem is that if I want to relocate a certain
> node (maybe an automatic relocation because of a hardware failure), I
> need to create the core manually in the new node because it's
> expecting to find the core.properties file in the data folder and of
> course it won't because the storage is ephemeral. Is there a way to
> make a new node join the cluster with no manual intervention ?
The things you're asking sound like SolrCloud. The rest of this message
assumes that you're running cloud. If you're not, then we may need to
start over.
When you start a new node, it automatically joins the cluster described
by the Zookeeper database that you point it to.
SolrCloud will **NOT** automatically create replicas when a new node
joins the cluster. There's no way for SolrCloud to know what you
actually want to use that new node for, so anything that it did
automatically might be completely the wrong thing.
Once you add a new node, you can replicate existing data to it with the
ADDREPLICA action on the Collections API:
https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api_addreplica
If the original problem was a down node, you might also want to use the
DELETEREPLICA action to delete any replicas on the node that you lost
that are marked down:
https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api9
Creating cores manually in your situation is not advisable. The
CoreAdmin API should not be used when you're running in cloud mode.
Thanks,
Shawn