You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Lorenzo Fundaró <lo...@dawandamail.com> on 2016/07/04 16:18:46 UTC
deploy solr on cloud providers
Hi guys,
when deploying solr (in solrcloud mode) in the cloud one has to take care
of storage, and as far as I understand it can be a problem because the
storage should go wherever the node is created. If we have for example, a
node on EC2 with its own persistent disk, this node happens to be the
leader and at some point crashes but couldn't make the replication of the
data that has in the transaction log, how do we do in that case ? Ideally
the new node must use the leftover data that the death node left, but this
is a bit cumbersome in my opinion.
What are the best practices for this ?
thanks in advance.
--
--
Lorenzo Fundaro
Backend Engineer
E-Mail: lorenzo.fundaro@dawandamail.com
Fax + 49 - (0)30 - 25 76 08 52
Tel + 49 - (0)179 - 51 10 982
DaWanda GmbH
Windscheidstraße 18
10627 Berlin
Geschäftsführer: Claudia Helming und Niels Nüssler
AG Charlottenburg HRB 104695 B http://www.dawanda.com
Re: deploy solr on cloud providers
Posted by Tomás Fernández Löbbe <to...@gmail.com>.
I think there are two parts to this question:
* If a node just disappears you should be fine in terms of data
availability, since Solr in "SolrCloud" replicates the data as it comes it
(before sending the http response). Even if the leader disappears and never
comes back as long as you have one replica alive for that shard of that
collection there should be no data lost. A new leader will be elected and
you can continue adding docs or querying.
* If the node doesn't recover and a new one joins the cluster, currently
Solr won't automatically realize that replicas have disappear and create
them, so you need to take some action. Some good responses about this issue
are in this other thread
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201607.mbox/%3CCAP_WmbugkdUjiN1unB_ArvxQ9vh3F5x6yBpgU7iQCkAWV9bfLQ@mail.gmail.com%3E
I hope this helps,
Tomás
On Tue, Jul 5, 2016 at 8:55 AM, Steven Bower <sb...@alcyon.net> wrote:
> You shouldn't "need" to move the storage as SolrCloud will replicate all
> data to the new node and anything in the transaction log will already be
> distributed through the rest of the machines..
>
> One option to keep all your data attached to nodes might be to use Amazon
> EFS (pretty new) to store your data.. However I've not seen any good perf
> testing done against it so not sure how it will scale..
>
> steve
>
> On Tue, Jul 5, 2016 at 11:46 AM Lorenzo Fundaró <
> lorenzo.fundaro@dawandamail.com> wrote:
>
> > On 5 July 2016 at 15:55, Shawn Heisey <ap...@elyograg.org> wrote:
> >
> > > On 7/5/2016 1:19 AM, Lorenzo Fundaró wrote:
> > > > Hi Shawn. Actually what im trying to find out is whether this is the
> > best
> > > > approach for deploying solr in the cloud. I believe solrcloud solves
> a
> > > lot
> > > > of problems in terms of High Availability but when it comes to
> storage
> > > > there seems to be a limitation that can be workaround of course but
> > it's
> > > a
> > > > bit cumbersome and i was wondering if there is a better option for
> this
> > > or
> > > > if im missing something with the way I'm doing it. I wonder if there
> > are
> > > > some proved experience about how to solve the storage problem when
> > > > deploying in the cloud. Any advise or point to some enlightening
> > > > documentation will be appreciated. Thanks.
> > >
> > > When you ask whether "this is the best approach" ... you need to define
> > > what "this" is. You mention a "storage problem" that needs solving ...
> > > but haven't actually described that problem in a way that I can
> > > understand.
> >
> >
> > So, Im trying to put Solrcloud in a cloud provider where a node can
> > disappear any time
> > because of hardware failure. In order to preserve any non replicated
> > updates I need to
> > make the storage of that dead node go to the newly spawned node. I am not
> > having a problem with this
> > approach actually, I just want to know if there is a better way of doing
> > this. I know there is HDFS support that makes
> > all this easier but this is not an option for me. Thank you and I
> apologise
> > for the unclear mails.
> >
> >
> > >
> > > Let's back up and cover some basics:
> > >
> > > What steps are you taking?
> >
> > What do you expect (or want) to happen?
> >
> > What actually happens?
> > >
> > > The answers to these questions need to be very detailed.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
> >
> > --
> >
> > --
> > Lorenzo Fundaro
> > Backend Engineer
> > E-Mail: lorenzo.fundaro@dawandamail.com
> >
> > Fax + 49 - (0)30 - 25 76 08 52
> > Tel + 49 - (0)179 - 51 10 982
> >
> > DaWanda GmbH
> > Windscheidstraße 18
> > 10627 Berlin
> >
> > Geschäftsführer: Claudia Helming und Niels Nüssler
> > AG Charlottenburg HRB 104695 B http://www.dawanda.com
> >
>
Re: deploy solr on cloud providers
Posted by Steven Bower <sb...@alcyon.net>.
You shouldn't "need" to move the storage as SolrCloud will replicate all
data to the new node and anything in the transaction log will already be
distributed through the rest of the machines..
One option to keep all your data attached to nodes might be to use Amazon
EFS (pretty new) to store your data.. However I've not seen any good perf
testing done against it so not sure how it will scale..
steve
On Tue, Jul 5, 2016 at 11:46 AM Lorenzo Fundaró <
lorenzo.fundaro@dawandamail.com> wrote:
> On 5 July 2016 at 15:55, Shawn Heisey <ap...@elyograg.org> wrote:
>
> > On 7/5/2016 1:19 AM, Lorenzo Fundaró wrote:
> > > Hi Shawn. Actually what im trying to find out is whether this is the
> best
> > > approach for deploying solr in the cloud. I believe solrcloud solves a
> > lot
> > > of problems in terms of High Availability but when it comes to storage
> > > there seems to be a limitation that can be workaround of course but
> it's
> > a
> > > bit cumbersome and i was wondering if there is a better option for this
> > or
> > > if im missing something with the way I'm doing it. I wonder if there
> are
> > > some proved experience about how to solve the storage problem when
> > > deploying in the cloud. Any advise or point to some enlightening
> > > documentation will be appreciated. Thanks.
> >
> > When you ask whether "this is the best approach" ... you need to define
> > what "this" is. You mention a "storage problem" that needs solving ...
> > but haven't actually described that problem in a way that I can
> > understand.
>
>
> So, Im trying to put Solrcloud in a cloud provider where a node can
> disappear any time
> because of hardware failure. In order to preserve any non replicated
> updates I need to
> make the storage of that dead node go to the newly spawned node. I am not
> having a problem with this
> approach actually, I just want to know if there is a better way of doing
> this. I know there is HDFS support that makes
> all this easier but this is not an option for me. Thank you and I apologise
> for the unclear mails.
>
>
> >
> > Let's back up and cover some basics:
> >
> > What steps are you taking?
>
> What do you expect (or want) to happen?
>
> What actually happens?
> >
> > The answers to these questions need to be very detailed.
> >
> > Thanks,
> > Shawn
> >
> >
>
>
> --
>
> --
> Lorenzo Fundaro
> Backend Engineer
> E-Mail: lorenzo.fundaro@dawandamail.com
>
> Fax + 49 - (0)30 - 25 76 08 52
> Tel + 49 - (0)179 - 51 10 982
>
> DaWanda GmbH
> Windscheidstraße 18
> 10627 Berlin
>
> Geschäftsführer: Claudia Helming und Niels Nüssler
> AG Charlottenburg HRB 104695 B http://www.dawanda.com
>
Re: deploy solr on cloud providers
Posted by Lorenzo Fundaró <lo...@dawandamail.com>.
On 5 July 2016 at 15:55, Shawn Heisey <ap...@elyograg.org> wrote:
> On 7/5/2016 1:19 AM, Lorenzo Fundaró wrote:
> > Hi Shawn. Actually what im trying to find out is whether this is the best
> > approach for deploying solr in the cloud. I believe solrcloud solves a
> lot
> > of problems in terms of High Availability but when it comes to storage
> > there seems to be a limitation that can be workaround of course but it's
> a
> > bit cumbersome and i was wondering if there is a better option for this
> or
> > if im missing something with the way I'm doing it. I wonder if there are
> > some proved experience about how to solve the storage problem when
> > deploying in the cloud. Any advise or point to some enlightening
> > documentation will be appreciated. Thanks.
>
> When you ask whether "this is the best approach" ... you need to define
> what "this" is. You mention a "storage problem" that needs solving ...
> but haven't actually described that problem in a way that I can
> understand.
So, Im trying to put Solrcloud in a cloud provider where a node can
disappear any time
because of hardware failure. In order to preserve any non replicated
updates I need to
make the storage of that dead node go to the newly spawned node. I am not
having a problem with this
approach actually, I just want to know if there is a better way of doing
this. I know there is HDFS support that makes
all this easier but this is not an option for me. Thank you and I apologise
for the unclear mails.
>
> Let's back up and cover some basics:
>
> What steps are you taking?
What do you expect (or want) to happen?
What actually happens?
>
> The answers to these questions need to be very detailed.
>
> Thanks,
> Shawn
>
>
--
--
Lorenzo Fundaro
Backend Engineer
E-Mail: lorenzo.fundaro@dawandamail.com
Fax + 49 - (0)30 - 25 76 08 52
Tel + 49 - (0)179 - 51 10 982
DaWanda GmbH
Windscheidstraße 18
10627 Berlin
Geschäftsführer: Claudia Helming und Niels Nüssler
AG Charlottenburg HRB 104695 B http://www.dawanda.com
Re: deploy solr on cloud providers
Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/5/2016 1:19 AM, Lorenzo Fundar� wrote:
> Hi Shawn. Actually what im trying to find out is whether this is the best
> approach for deploying solr in the cloud. I believe solrcloud solves a lot
> of problems in terms of High Availability but when it comes to storage
> there seems to be a limitation that can be workaround of course but it's a
> bit cumbersome and i was wondering if there is a better option for this or
> if im missing something with the way I'm doing it. I wonder if there are
> some proved experience about how to solve the storage problem when
> deploying in the cloud. Any advise or point to some enlightening
> documentation will be appreciated. Thanks.
When you ask whether "this is the best approach" ... you need to define
what "this" is. You mention a "storage problem" that needs solving ...
but haven't actually described that problem in a way that I can understand.
Let's back up and cover some basics:
What steps are you taking?
What do you expect (or want) to happen?
What actually happens?
The answers to these questions need to be very detailed.
Thanks,
Shawn
Re: deploy solr on cloud providers
Posted by Steven Bower <sb...@alcyon.net>.
Looking deeper into zookeeper as truth mode I was wrong about existing
replicas being recreated once storage is gone.. Seems there is intent for
the type of behavior based upon existing tickets.. We'll look at creating a
patch for this too..
Steve
On Tue, Jul 5, 2016 at 6:00 PM Tomás Fernández Löbbe <to...@gmail.com>
wrote:
> The leader will do the replication before responding to the client, so lets
> say the leader gets to update it's local copy, but it's terminated before
> sending the request to the replicas, the client should get either an HTTP
> 500 or no http response. From the client code you can take action (log,
> retry, etc).
> The "min_rf" is useful for the case where replicas may be down or not
> accessible. Again, you can use this for retrying or take any necessary
> action on the client side if the desired rf is not achieved.
>
> Tomás
>
> On Tue, Jul 5, 2016 at 11:39 AM, Lorenzo Fundaró <
> lorenzo.fundaro@dawandamail.com> wrote:
>
> > @Tomas and @Steven
> >
> > I am a bit skeptical about this two statements:
> >
> > If a node just disappears you should be fine in terms of data
> > > availability, since Solr in "SolrCloud" replicates the data as it comes
> > it
> > > (before sending the http response)
> >
> >
> > and
> >
> > >
> > > You shouldn't "need" to move the storage as SolrCloud will replicate
> all
> > > data to the new node and anything in the transaction log will already
> be
> > > distributed through the rest of the machines..
> >
> >
> > because according to the official documentation here
> > <
> >
> https://cwiki.apache.org/confluence/display/solr/Read+and+Write+Side+Fault+Tolerance
> > >:
> > (Write side fault tolerant -> recovery)
> >
> > If a leader goes down, it may have sent requests to some replicas and not
> > > others. So when a new potential leader is identified, it runs a synch
> > > process against the other replicas. If this is successful, everything
> > > should be consistent, the leader registers as active, and normal
> actions
> > > proceed
> >
> >
> > I think there is a possibility that an update is not sent by the leader
> but
> > is kept in the local disk and after it comes up again it can sync the
> > non-sent data.
> >
> > Furthermore:
> >
> > Achieved Replication Factor
> > > When using a replication factor greater than one, an update request may
> > > succeed on the shard leader but fail on one or more of the replicas.
> For
> > > instance, consider a collection with one shard and replication factor
> of
> > > three. In this case, you have a shard leader and two additional
> replicas.
> > > If an update request succeeds on the leader but fails on both replicas,
> > for
> > > whatever reason, the update request is still considered successful from
> > the
> > > perspective of the client. The replicas that missed the update will
> sync
> > > with the leader when they recover.
> >
> >
> > They have implemented this parameter called *min_rf* that you can use
> > (client-side) to make sure that your update was replicated to at least
> one
> > replica (e.g.: min_rf > 1).
> >
> > This is why my concern about moving storage around, because then I know
> > when the shard leader comes back, solrcloud will run sync process for
> those
> > documents that couldn't be sent to the replicas.
> >
> > Am I missing something or misunderstood the documentation ?
> >
> > Cheers !
> >
> >
> >
> >
> >
> >
> >
> > On 5 July 2016 at 19:49, Davis, Daniel (NIH/NLM) [C] <
> daniel.davis@nih.gov
> > >
> > wrote:
> >
> > > Lorenzo, this probably comes late, but my systems guys just don't want
> to
> > > give me real disk. Although RAID-5 or LVM on-top of JBOD may be
> better
> > > than Amazon EBS, Amazon EBS is still much closer to real disk in terms
> of
> > > IOPS and latency than NFS ;) I even ran a mini test (not an official
> > > benchmark), and found the response time for random reads to be better.
> > >
> > > If you are a young/smallish company, this may be all in the cloud, but
> if
> > > you are in a large organization like mine, you may also need to allow
> for
> > > other architectures, such as a "virtual" Netapp in the cloud that
> > > communicates with a physical Netapp on-premises, and the
> > throughput/latency
> > > of that. The most important thing is to actually measure the numbers
> > you
> > > are getting, both for search and for simply raw I/O, or to get your
> > > systems/storage guys to measure those numbers. If you get your
> > > systems/storage guys to just measure storage - you will want to care
> > about
> > > three things for indexing primarily:
> > >
> > > Sequential Write Throughput
> > > Random Read Throughput
> > > Random Read Response Time/Latency
> > >
> > > Hope this helps,
> > >
> > > Dan Davis, Systems/Applications Architect (Contractor),
> > > Office of Computer and Communications Systems,
> > > National Library of Medicine, NIH
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Lorenzo Fundaró [mailto:lorenzo.fundaro@dawandamail.com]
> > > Sent: Tuesday, July 05, 2016 3:20 AM
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: deploy solr on cloud providers
> > >
> > > Hi Shawn. Actually what im trying to find out is whether this is the
> best
> > > approach for deploying solr in the cloud. I believe solrcloud solves a
> > lot
> > > of problems in terms of High Availability but when it comes to storage
> > > there seems to be a limitation that can be workaround of course but
> it's
> > a
> > > bit cumbersome and i was wondering if there is a better option for this
> > or
> > > if im missing something with the way I'm doing it. I wonder if there
> are
> > > some proved experience about how to solve the storage problem when
> > > deploying in the cloud. Any advise or point to some enlightening
> > > documentation will be appreciated. Thanks.
> > > On Jul 4, 2016 18:27, "Shawn Heisey" <ap...@elyograg.org> wrote:
> > >
> > > > On 7/4/2016 10:18 AM, Lorenzo Fundaró wrote:
> > > > > when deploying solr (in solrcloud mode) in the cloud one has to
> take
> > > > > care of storage, and as far as I understand it can be a problem
> > > > > because the storage should go wherever the node is created. If we
> > > > > have for example, a node on EC2 with its own persistent disk, this
> > > > > node happens to be the leader and at some point crashes but
> couldn't
> > > > > make the replication of the data that has in the transaction log,
> > > > > how do we do in that case ? Ideally the new node must use the
> > > > > leftover data that the death node left, but this is a bit
> cumbersome
> > > > > in my opinion. What are the best practices for this ?
> > > >
> > > > I can't make any sense of this. What is the *exact* problem you need
> > > > to solve? The details can be very important.
> > > >
> > > > We might be dealing with this:
> > > >
> > > > http://people.apache.org/~hossman/#xyproblem
> > > >
> > > > Thanks,
> > > > Shawn
> > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > --
> > Lorenzo Fundaro
> > Backend Engineer
> > E-Mail: lorenzo.fundaro@dawandamail.com
> >
> > Fax + 49 - (0)30 - 25 76 08 52
> > Tel + 49 - (0)179 - 51 10 982
> >
> > DaWanda GmbH
> > Windscheidstraße 18
> > 10627 Berlin
> >
> > Geschäftsführer: Claudia Helming und Niels Nüssler
> > AG Charlottenburg HRB 104695 B http://www.dawanda.com
> >
>
Re: deploy solr on cloud providers
Posted by Lorenzo Fundaró <lo...@dawandamail.com>.
Thank you Tomas, I would take a thorough look to the jira ticket you're
pointing out.
On 6 July 2016 at 20:49, Tomás Fernández Löbbe <to...@gmail.com>
wrote:
> On Wed, Jul 6, 2016 at 2:30 AM, Lorenzo Fundaró <
> lorenzo.fundaro@dawandamail.com> wrote:
>
> > On 6 July 2016 at 00:00, Tomás Fernández Löbbe <to...@gmail.com>
> > wrote:
> >
> > > The leader will do the replication before responding to the client, so
> > lets
> > > say the leader gets to update it's local copy, but it's terminated
> before
> > > sending the request to the replicas, the client should get either an
> HTTP
> > > 500 or no http response. From the client code you can take action (log,
> > > retry, etc).
> > >
> >
> > If this true then whenever I ask for min_rf having three nodes (1 leader
> +
> > 2 replicas)
> > I should get rf = 3, but in reality i don't.
> >
> >
> > > The "min_rf" is useful for the case where replicas may be down or not
> > > accessible. Again, you can use this for retrying or take any necessary
> > > action on the client side if the desired rf is not achieved.
> > >
> >
> >
> > I think both paragraphs are contradictory. If the leader does the
> > replication before responding to the client, then
> > why is there a need to use the min_rf ? I don;t think is true that you
> get
> > a 200 when the update has been passed to all replicas.
> >
>
> The reason why "min_rf" is there is because:
> * If there are no replicas at the time of the request (e.g. if replicas are
> unreachable and disconnected from ZK)
> * Replicas could fail to ACK the update request from the leader, in that
> case the leader will mark them as unhealthy but would HTTP 200 to the
> client.
>
> So, it could happen that you think your data is being replicated to 3
> replicas, but 2 of them are currently out of service, this means that your
> doc is in a single host, and if that one dies, then you lose that data. In
> order to prevent this, you can ask Solr to tell you how many replicas
> succeeded that update request. You can read more about this in
> https://issues.apache.org/jira/browse/SOLR-5468
>
>
> >
> > The thing is that, when you have persistent storage yo shouldn't worry
> > about this because you know when the node comes back
> > the rest of the index will be sync, the problem is when you don't have
> > persistent storage. For my particular case I have to be extra careful and
> > always
> > make sure that all my replicas have all the data I sent.
> >
> > In any case you should assume that storage on a host can be completely
> lost, no mater if you are deploying on premises or on the cloud. Consider
> that once that host comes back (could be hours later) it could be already
> out of date, and will replicate from the current leader, possibly dropping
> parts or all it's current index.
>
> Tomás
>
>
> >
> > > Tomás
> > >
> > > On Tue, Jul 5, 2016 at 11:39 AM, Lorenzo Fundaró <
> > > lorenzo.fundaro@dawandamail.com> wrote:
> > >
> > > > @Tomas and @Steven
> > > >
> > > > I am a bit skeptical about this two statements:
> > > >
> > > > If a node just disappears you should be fine in terms of data
> > > > > availability, since Solr in "SolrCloud" replicates the data as it
> > comes
> > > > it
> > > > > (before sending the http response)
> > > >
> > > >
> > > > and
> > > >
> > > > >
> > > > > You shouldn't "need" to move the storage as SolrCloud will
> replicate
> > > all
> > > > > data to the new node and anything in the transaction log will
> already
> > > be
> > > > > distributed through the rest of the machines..
> > > >
> > > >
> > > > because according to the official documentation here
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/solr/Read+and+Write+Side+Fault+Tolerance
> > > > >:
> > > > (Write side fault tolerant -> recovery)
> > > >
> > > > If a leader goes down, it may have sent requests to some replicas and
> > not
> > > > > others. So when a new potential leader is identified, it runs a
> synch
> > > > > process against the other replicas. If this is successful,
> everything
> > > > > should be consistent, the leader registers as active, and normal
> > > actions
> > > > > proceed
> > > >
> > > >
> > > > I think there is a possibility that an update is not sent by the
> leader
> > > but
> > > > is kept in the local disk and after it comes up again it can sync the
> > > > non-sent data.
> > > >
> > > > Furthermore:
> > > >
> > > > Achieved Replication Factor
> > > > > When using a replication factor greater than one, an update request
> > may
> > > > > succeed on the shard leader but fail on one or more of the
> replicas.
> > > For
> > > > > instance, consider a collection with one shard and replication
> factor
> > > of
> > > > > three. In this case, you have a shard leader and two additional
> > > replicas.
> > > > > If an update request succeeds on the leader but fails on both
> > replicas,
> > > > for
> > > > > whatever reason, the update request is still considered successful
> > from
> > > > the
> > > > > perspective of the client. The replicas that missed the update will
> > > sync
> > > > > with the leader when they recover.
> > > >
> > > >
> > > > They have implemented this parameter called *min_rf* that you can use
> > > > (client-side) to make sure that your update was replicated to at
> least
> > > one
> > > > replica (e.g.: min_rf > 1).
> > > >
> > > > This is why my concern about moving storage around, because then I
> know
> > > > when the shard leader comes back, solrcloud will run sync process for
> > > those
> > > > documents that couldn't be sent to the replicas.
> > > >
> > > > Am I missing something or misunderstood the documentation ?
> > > >
> > > > Cheers !
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 5 July 2016 at 19:49, Davis, Daniel (NIH/NLM) [C] <
> > > daniel.davis@nih.gov
> > > > >
> > > > wrote:
> > > >
> > > > > Lorenzo, this probably comes late, but my systems guys just don't
> > want
> > > to
> > > > > give me real disk. Although RAID-5 or LVM on-top of JBOD may be
> > > better
> > > > > than Amazon EBS, Amazon EBS is still much closer to real disk in
> > terms
> > > of
> > > > > IOPS and latency than NFS ;) I even ran a mini test (not an
> > official
> > > > > benchmark), and found the response time for random reads to be
> > better.
> > > > >
> > > > > If you are a young/smallish company, this may be all in the cloud,
> > but
> > > if
> > > > > you are in a large organization like mine, you may also need to
> allow
> > > for
> > > > > other architectures, such as a "virtual" Netapp in the cloud that
> > > > > communicates with a physical Netapp on-premises, and the
> > > > throughput/latency
> > > > > of that. The most important thing is to actually measure the
> > numbers
> > > > you
> > > > > are getting, both for search and for simply raw I/O, or to get your
> > > > > systems/storage guys to measure those numbers. If you get your
> > > > > systems/storage guys to just measure storage - you will want to
> care
> > > > about
> > > > > three things for indexing primarily:
> > > > >
> > > > > Sequential Write Throughput
> > > > > Random Read Throughput
> > > > > Random Read Response Time/Latency
> > > > >
> > > > > Hope this helps,
> > > > >
> > > > > Dan Davis, Systems/Applications Architect (Contractor),
> > > > > Office of Computer and Communications Systems,
> > > > > National Library of Medicine, NIH
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Lorenzo Fundaró [mailto:lorenzo.fundaro@dawandamail.com]
> > > > > Sent: Tuesday, July 05, 2016 3:20 AM
> > > > > To: solr-user@lucene.apache.org
> > > > > Subject: Re: deploy solr on cloud providers
> > > > >
> > > > > Hi Shawn. Actually what im trying to find out is whether this is
> the
> > > best
> > > > > approach for deploying solr in the cloud. I believe solrcloud
> solves
> > a
> > > > lot
> > > > > of problems in terms of High Availability but when it comes to
> > storage
> > > > > there seems to be a limitation that can be workaround of course but
> > > it's
> > > > a
> > > > > bit cumbersome and i was wondering if there is a better option for
> > this
> > > > or
> > > > > if im missing something with the way I'm doing it. I wonder if
> there
> > > are
> > > > > some proved experience about how to solve the storage problem when
> > > > > deploying in the cloud. Any advise or point to some enlightening
> > > > > documentation will be appreciated. Thanks.
> > > > > On Jul 4, 2016 18:27, "Shawn Heisey" <ap...@elyograg.org> wrote:
> > > > >
> > > > > > On 7/4/2016 10:18 AM, Lorenzo Fundaró wrote:
> > > > > > > when deploying solr (in solrcloud mode) in the cloud one has to
> > > take
> > > > > > > care of storage, and as far as I understand it can be a problem
> > > > > > > because the storage should go wherever the node is created. If
> we
> > > > > > > have for example, a node on EC2 with its own persistent disk,
> > this
> > > > > > > node happens to be the leader and at some point crashes but
> > > couldn't
> > > > > > > make the replication of the data that has in the transaction
> log,
> > > > > > > how do we do in that case ? Ideally the new node must use the
> > > > > > > leftover data that the death node left, but this is a bit
> > > cumbersome
> > > > > > > in my opinion. What are the best practices for this ?
> > > > > >
> > > > > > I can't make any sense of this. What is the *exact* problem you
> > need
> > > > > > to solve? The details can be very important.
> > > > > >
> > > > > > We might be dealing with this:
> > > > > >
> > > > > > http://people.apache.org/~hossman/#xyproblem
> > > > > >
> > > > > > Thanks,
> > > > > > Shawn
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > --
> > > > Lorenzo Fundaro
> > > > Backend Engineer
> > > > E-Mail: lorenzo.fundaro@dawandamail.com
> > > >
> > > > Fax + 49 - (0)30 - 25 76 08 52
> > > > Tel + 49 - (0)179 - 51 10 982
> > > >
> > > > DaWanda GmbH
> > > > Windscheidstraße 18
> > > > 10627 Berlin
> > > >
> > > > Geschäftsführer: Claudia Helming und Niels Nüssler
> > > > AG Charlottenburg HRB 104695 B http://www.dawanda.com
> > > >
> > >
> >
> >
> >
> > --
> >
> > --
> > Lorenzo Fundaro
> > Backend Engineer
> > E-Mail: lorenzo.fundaro@dawandamail.com
> >
> > Fax + 49 - (0)30 - 25 76 08 52
> > Tel + 49 - (0)179 - 51 10 982
> >
> > DaWanda GmbH
> > Windscheidstraße 18
> > 10627 Berlin
> >
> > Geschäftsführer: Claudia Helming und Niels Nüssler
> > AG Charlottenburg HRB 104695 B http://www.dawanda.com
> >
>
--
--
Lorenzo Fundaro
Backend Engineer
E-Mail: lorenzo.fundaro@dawandamail.com
Fax + 49 - (0)30 - 25 76 08 52
Tel + 49 - (0)179 - 51 10 982
DaWanda GmbH
Windscheidstraße 18
10627 Berlin
Geschäftsführer: Claudia Helming und Niels Nüssler
AG Charlottenburg HRB 104695 B http://www.dawanda.com
Re: deploy solr on cloud providers
Posted by Tomás Fernández Löbbe <to...@gmail.com>.
On Wed, Jul 6, 2016 at 2:30 AM, Lorenzo Fundaró <
lorenzo.fundaro@dawandamail.com> wrote:
> On 6 July 2016 at 00:00, Tomás Fernández Löbbe <to...@gmail.com>
> wrote:
>
> > The leader will do the replication before responding to the client, so
> lets
> > say the leader gets to update it's local copy, but it's terminated before
> > sending the request to the replicas, the client should get either an HTTP
> > 500 or no http response. From the client code you can take action (log,
> > retry, etc).
> >
>
> If this true then whenever I ask for min_rf having three nodes (1 leader +
> 2 replicas)
> I should get rf = 3, but in reality i don't.
>
>
> > The "min_rf" is useful for the case where replicas may be down or not
> > accessible. Again, you can use this for retrying or take any necessary
> > action on the client side if the desired rf is not achieved.
> >
>
>
> I think both paragraphs are contradictory. If the leader does the
> replication before responding to the client, then
> why is there a need to use the min_rf ? I don;t think is true that you get
> a 200 when the update has been passed to all replicas.
>
The reason why "min_rf" is there is because:
* If there are no replicas at the time of the request (e.g. if replicas are
unreachable and disconnected from ZK)
* Replicas could fail to ACK the update request from the leader, in that
case the leader will mark them as unhealthy but would HTTP 200 to the
client.
So, it could happen that you think your data is being replicated to 3
replicas, but 2 of them are currently out of service, this means that your
doc is in a single host, and if that one dies, then you lose that data. In
order to prevent this, you can ask Solr to tell you how many replicas
succeeded that update request. You can read more about this in
https://issues.apache.org/jira/browse/SOLR-5468
>
> The thing is that, when you have persistent storage yo shouldn't worry
> about this because you know when the node comes back
> the rest of the index will be sync, the problem is when you don't have
> persistent storage. For my particular case I have to be extra careful and
> always
> make sure that all my replicas have all the data I sent.
>
> In any case you should assume that storage on a host can be completely
lost, no mater if you are deploying on premises or on the cloud. Consider
that once that host comes back (could be hours later) it could be already
out of date, and will replicate from the current leader, possibly dropping
parts or all it's current index.
Tomás
>
> > Tomás
> >
> > On Tue, Jul 5, 2016 at 11:39 AM, Lorenzo Fundaró <
> > lorenzo.fundaro@dawandamail.com> wrote:
> >
> > > @Tomas and @Steven
> > >
> > > I am a bit skeptical about this two statements:
> > >
> > > If a node just disappears you should be fine in terms of data
> > > > availability, since Solr in "SolrCloud" replicates the data as it
> comes
> > > it
> > > > (before sending the http response)
> > >
> > >
> > > and
> > >
> > > >
> > > > You shouldn't "need" to move the storage as SolrCloud will replicate
> > all
> > > > data to the new node and anything in the transaction log will already
> > be
> > > > distributed through the rest of the machines..
> > >
> > >
> > > because according to the official documentation here
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/solr/Read+and+Write+Side+Fault+Tolerance
> > > >:
> > > (Write side fault tolerant -> recovery)
> > >
> > > If a leader goes down, it may have sent requests to some replicas and
> not
> > > > others. So when a new potential leader is identified, it runs a synch
> > > > process against the other replicas. If this is successful, everything
> > > > should be consistent, the leader registers as active, and normal
> > actions
> > > > proceed
> > >
> > >
> > > I think there is a possibility that an update is not sent by the leader
> > but
> > > is kept in the local disk and after it comes up again it can sync the
> > > non-sent data.
> > >
> > > Furthermore:
> > >
> > > Achieved Replication Factor
> > > > When using a replication factor greater than one, an update request
> may
> > > > succeed on the shard leader but fail on one or more of the replicas.
> > For
> > > > instance, consider a collection with one shard and replication factor
> > of
> > > > three. In this case, you have a shard leader and two additional
> > replicas.
> > > > If an update request succeeds on the leader but fails on both
> replicas,
> > > for
> > > > whatever reason, the update request is still considered successful
> from
> > > the
> > > > perspective of the client. The replicas that missed the update will
> > sync
> > > > with the leader when they recover.
> > >
> > >
> > > They have implemented this parameter called *min_rf* that you can use
> > > (client-side) to make sure that your update was replicated to at least
> > one
> > > replica (e.g.: min_rf > 1).
> > >
> > > This is why my concern about moving storage around, because then I know
> > > when the shard leader comes back, solrcloud will run sync process for
> > those
> > > documents that couldn't be sent to the replicas.
> > >
> > > Am I missing something or misunderstood the documentation ?
> > >
> > > Cheers !
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 5 July 2016 at 19:49, Davis, Daniel (NIH/NLM) [C] <
> > daniel.davis@nih.gov
> > > >
> > > wrote:
> > >
> > > > Lorenzo, this probably comes late, but my systems guys just don't
> want
> > to
> > > > give me real disk. Although RAID-5 or LVM on-top of JBOD may be
> > better
> > > > than Amazon EBS, Amazon EBS is still much closer to real disk in
> terms
> > of
> > > > IOPS and latency than NFS ;) I even ran a mini test (not an
> official
> > > > benchmark), and found the response time for random reads to be
> better.
> > > >
> > > > If you are a young/smallish company, this may be all in the cloud,
> but
> > if
> > > > you are in a large organization like mine, you may also need to allow
> > for
> > > > other architectures, such as a "virtual" Netapp in the cloud that
> > > > communicates with a physical Netapp on-premises, and the
> > > throughput/latency
> > > > of that. The most important thing is to actually measure the
> numbers
> > > you
> > > > are getting, both for search and for simply raw I/O, or to get your
> > > > systems/storage guys to measure those numbers. If you get your
> > > > systems/storage guys to just measure storage - you will want to care
> > > about
> > > > three things for indexing primarily:
> > > >
> > > > Sequential Write Throughput
> > > > Random Read Throughput
> > > > Random Read Response Time/Latency
> > > >
> > > > Hope this helps,
> > > >
> > > > Dan Davis, Systems/Applications Architect (Contractor),
> > > > Office of Computer and Communications Systems,
> > > > National Library of Medicine, NIH
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Lorenzo Fundaró [mailto:lorenzo.fundaro@dawandamail.com]
> > > > Sent: Tuesday, July 05, 2016 3:20 AM
> > > > To: solr-user@lucene.apache.org
> > > > Subject: Re: deploy solr on cloud providers
> > > >
> > > > Hi Shawn. Actually what im trying to find out is whether this is the
> > best
> > > > approach for deploying solr in the cloud. I believe solrcloud solves
> a
> > > lot
> > > > of problems in terms of High Availability but when it comes to
> storage
> > > > there seems to be a limitation that can be workaround of course but
> > it's
> > > a
> > > > bit cumbersome and i was wondering if there is a better option for
> this
> > > or
> > > > if im missing something with the way I'm doing it. I wonder if there
> > are
> > > > some proved experience about how to solve the storage problem when
> > > > deploying in the cloud. Any advise or point to some enlightening
> > > > documentation will be appreciated. Thanks.
> > > > On Jul 4, 2016 18:27, "Shawn Heisey" <ap...@elyograg.org> wrote:
> > > >
> > > > > On 7/4/2016 10:18 AM, Lorenzo Fundaró wrote:
> > > > > > when deploying solr (in solrcloud mode) in the cloud one has to
> > take
> > > > > > care of storage, and as far as I understand it can be a problem
> > > > > > because the storage should go wherever the node is created. If we
> > > > > > have for example, a node on EC2 with its own persistent disk,
> this
> > > > > > node happens to be the leader and at some point crashes but
> > couldn't
> > > > > > make the replication of the data that has in the transaction log,
> > > > > > how do we do in that case ? Ideally the new node must use the
> > > > > > leftover data that the death node left, but this is a bit
> > cumbersome
> > > > > > in my opinion. What are the best practices for this ?
> > > > >
> > > > > I can't make any sense of this. What is the *exact* problem you
> need
> > > > > to solve? The details can be very important.
> > > > >
> > > > > We might be dealing with this:
> > > > >
> > > > > http://people.apache.org/~hossman/#xyproblem
> > > > >
> > > > > Thanks,
> > > > > Shawn
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > --
> > > Lorenzo Fundaro
> > > Backend Engineer
> > > E-Mail: lorenzo.fundaro@dawandamail.com
> > >
> > > Fax + 49 - (0)30 - 25 76 08 52
> > > Tel + 49 - (0)179 - 51 10 982
> > >
> > > DaWanda GmbH
> > > Windscheidstraße 18
> > > 10627 Berlin
> > >
> > > Geschäftsführer: Claudia Helming und Niels Nüssler
> > > AG Charlottenburg HRB 104695 B http://www.dawanda.com
> > >
> >
>
>
>
> --
>
> --
> Lorenzo Fundaro
> Backend Engineer
> E-Mail: lorenzo.fundaro@dawandamail.com
>
> Fax + 49 - (0)30 - 25 76 08 52
> Tel + 49 - (0)179 - 51 10 982
>
> DaWanda GmbH
> Windscheidstraße 18
> 10627 Berlin
>
> Geschäftsführer: Claudia Helming und Niels Nüssler
> AG Charlottenburg HRB 104695 B http://www.dawanda.com
>
Re: deploy solr on cloud providers
Posted by Lorenzo Fundaró <lo...@dawandamail.com>.
On 6 July 2016 at 00:00, Tomás Fernández Löbbe <to...@gmail.com>
wrote:
> The leader will do the replication before responding to the client, so lets
> say the leader gets to update it's local copy, but it's terminated before
> sending the request to the replicas, the client should get either an HTTP
> 500 or no http response. From the client code you can take action (log,
> retry, etc).
>
If this true then whenever I ask for min_rf having three nodes (1 leader +
2 replicas)
I should get rf = 3, but in reality i don't.
> The "min_rf" is useful for the case where replicas may be down or not
> accessible. Again, you can use this for retrying or take any necessary
> action on the client side if the desired rf is not achieved.
>
I think both paragraphs are contradictory. If the leader does the
replication before responding to the client, then
why is there a need to use the min_rf ? I don;t think is true that you get
a 200 when the update has been passed to all replicas.
The thing is that, when you have persistent storage yo shouldn't worry
about this because you know when the node comes back
the rest of the index will be sync, the problem is when you don't have
persistent storage. For my particular case I have to be extra careful and
always
make sure that all my replicas have all the data I sent.
> Tomás
>
> On Tue, Jul 5, 2016 at 11:39 AM, Lorenzo Fundaró <
> lorenzo.fundaro@dawandamail.com> wrote:
>
> > @Tomas and @Steven
> >
> > I am a bit skeptical about this two statements:
> >
> > If a node just disappears you should be fine in terms of data
> > > availability, since Solr in "SolrCloud" replicates the data as it comes
> > it
> > > (before sending the http response)
> >
> >
> > and
> >
> > >
> > > You shouldn't "need" to move the storage as SolrCloud will replicate
> all
> > > data to the new node and anything in the transaction log will already
> be
> > > distributed through the rest of the machines..
> >
> >
> > because according to the official documentation here
> > <
> >
> https://cwiki.apache.org/confluence/display/solr/Read+and+Write+Side+Fault+Tolerance
> > >:
> > (Write side fault tolerant -> recovery)
> >
> > If a leader goes down, it may have sent requests to some replicas and not
> > > others. So when a new potential leader is identified, it runs a synch
> > > process against the other replicas. If this is successful, everything
> > > should be consistent, the leader registers as active, and normal
> actions
> > > proceed
> >
> >
> > I think there is a possibility that an update is not sent by the leader
> but
> > is kept in the local disk and after it comes up again it can sync the
> > non-sent data.
> >
> > Furthermore:
> >
> > Achieved Replication Factor
> > > When using a replication factor greater than one, an update request may
> > > succeed on the shard leader but fail on one or more of the replicas.
> For
> > > instance, consider a collection with one shard and replication factor
> of
> > > three. In this case, you have a shard leader and two additional
> replicas.
> > > If an update request succeeds on the leader but fails on both replicas,
> > for
> > > whatever reason, the update request is still considered successful from
> > the
> > > perspective of the client. The replicas that missed the update will
> sync
> > > with the leader when they recover.
> >
> >
> > They have implemented this parameter called *min_rf* that you can use
> > (client-side) to make sure that your update was replicated to at least
> one
> > replica (e.g.: min_rf > 1).
> >
> > This is why my concern about moving storage around, because then I know
> > when the shard leader comes back, solrcloud will run sync process for
> those
> > documents that couldn't be sent to the replicas.
> >
> > Am I missing something or misunderstood the documentation ?
> >
> > Cheers !
> >
> >
> >
> >
> >
> >
> >
> > On 5 July 2016 at 19:49, Davis, Daniel (NIH/NLM) [C] <
> daniel.davis@nih.gov
> > >
> > wrote:
> >
> > > Lorenzo, this probably comes late, but my systems guys just don't want
> to
> > > give me real disk. Although RAID-5 or LVM on-top of JBOD may be
> better
> > > than Amazon EBS, Amazon EBS is still much closer to real disk in terms
> of
> > > IOPS and latency than NFS ;) I even ran a mini test (not an official
> > > benchmark), and found the response time for random reads to be better.
> > >
> > > If you are a young/smallish company, this may be all in the cloud, but
> if
> > > you are in a large organization like mine, you may also need to allow
> for
> > > other architectures, such as a "virtual" Netapp in the cloud that
> > > communicates with a physical Netapp on-premises, and the
> > throughput/latency
> > > of that. The most important thing is to actually measure the numbers
> > you
> > > are getting, both for search and for simply raw I/O, or to get your
> > > systems/storage guys to measure those numbers. If you get your
> > > systems/storage guys to just measure storage - you will want to care
> > about
> > > three things for indexing primarily:
> > >
> > > Sequential Write Throughput
> > > Random Read Throughput
> > > Random Read Response Time/Latency
> > >
> > > Hope this helps,
> > >
> > > Dan Davis, Systems/Applications Architect (Contractor),
> > > Office of Computer and Communications Systems,
> > > National Library of Medicine, NIH
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Lorenzo Fundaró [mailto:lorenzo.fundaro@dawandamail.com]
> > > Sent: Tuesday, July 05, 2016 3:20 AM
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: deploy solr on cloud providers
> > >
> > > Hi Shawn. Actually what im trying to find out is whether this is the
> best
> > > approach for deploying solr in the cloud. I believe solrcloud solves a
> > lot
> > > of problems in terms of High Availability but when it comes to storage
> > > there seems to be a limitation that can be workaround of course but
> it's
> > a
> > > bit cumbersome and i was wondering if there is a better option for this
> > or
> > > if im missing something with the way I'm doing it. I wonder if there
> are
> > > some proved experience about how to solve the storage problem when
> > > deploying in the cloud. Any advise or point to some enlightening
> > > documentation will be appreciated. Thanks.
> > > On Jul 4, 2016 18:27, "Shawn Heisey" <ap...@elyograg.org> wrote:
> > >
> > > > On 7/4/2016 10:18 AM, Lorenzo Fundaró wrote:
> > > > > when deploying solr (in solrcloud mode) in the cloud one has to
> take
> > > > > care of storage, and as far as I understand it can be a problem
> > > > > because the storage should go wherever the node is created. If we
> > > > > have for example, a node on EC2 with its own persistent disk, this
> > > > > node happens to be the leader and at some point crashes but
> couldn't
> > > > > make the replication of the data that has in the transaction log,
> > > > > how do we do in that case ? Ideally the new node must use the
> > > > > leftover data that the death node left, but this is a bit
> cumbersome
> > > > > in my opinion. What are the best practices for this ?
> > > >
> > > > I can't make any sense of this. What is the *exact* problem you need
> > > > to solve? The details can be very important.
> > > >
> > > > We might be dealing with this:
> > > >
> > > > http://people.apache.org/~hossman/#xyproblem
> > > >
> > > > Thanks,
> > > > Shawn
> > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > --
> > Lorenzo Fundaro
> > Backend Engineer
> > E-Mail: lorenzo.fundaro@dawandamail.com
> >
> > Fax + 49 - (0)30 - 25 76 08 52
> > Tel + 49 - (0)179 - 51 10 982
> >
> > DaWanda GmbH
> > Windscheidstraße 18
> > 10627 Berlin
> >
> > Geschäftsführer: Claudia Helming und Niels Nüssler
> > AG Charlottenburg HRB 104695 B http://www.dawanda.com
> >
>
--
--
Lorenzo Fundaro
Backend Engineer
E-Mail: lorenzo.fundaro@dawandamail.com
Fax + 49 - (0)30 - 25 76 08 52
Tel + 49 - (0)179 - 51 10 982
DaWanda GmbH
Windscheidstraße 18
10627 Berlin
Geschäftsführer: Claudia Helming und Niels Nüssler
AG Charlottenburg HRB 104695 B http://www.dawanda.com
Re: deploy solr on cloud providers
Posted by Tomás Fernández Löbbe <to...@gmail.com>.
The leader will do the replication before responding to the client, so lets
say the leader gets to update it's local copy, but it's terminated before
sending the request to the replicas, the client should get either an HTTP
500 or no http response. From the client code you can take action (log,
retry, etc).
The "min_rf" is useful for the case where replicas may be down or not
accessible. Again, you can use this for retrying or take any necessary
action on the client side if the desired rf is not achieved.
Tomás
On Tue, Jul 5, 2016 at 11:39 AM, Lorenzo Fundaró <
lorenzo.fundaro@dawandamail.com> wrote:
> @Tomas and @Steven
>
> I am a bit skeptical about this two statements:
>
> If a node just disappears you should be fine in terms of data
> > availability, since Solr in "SolrCloud" replicates the data as it comes
> it
> > (before sending the http response)
>
>
> and
>
> >
> > You shouldn't "need" to move the storage as SolrCloud will replicate all
> > data to the new node and anything in the transaction log will already be
> > distributed through the rest of the machines..
>
>
> because according to the official documentation here
> <
> https://cwiki.apache.org/confluence/display/solr/Read+and+Write+Side+Fault+Tolerance
> >:
> (Write side fault tolerant -> recovery)
>
> If a leader goes down, it may have sent requests to some replicas and not
> > others. So when a new potential leader is identified, it runs a synch
> > process against the other replicas. If this is successful, everything
> > should be consistent, the leader registers as active, and normal actions
> > proceed
>
>
> I think there is a possibility that an update is not sent by the leader but
> is kept in the local disk and after it comes up again it can sync the
> non-sent data.
>
> Furthermore:
>
> Achieved Replication Factor
> > When using a replication factor greater than one, an update request may
> > succeed on the shard leader but fail on one or more of the replicas. For
> > instance, consider a collection with one shard and replication factor of
> > three. In this case, you have a shard leader and two additional replicas.
> > If an update request succeeds on the leader but fails on both replicas,
> for
> > whatever reason, the update request is still considered successful from
> the
> > perspective of the client. The replicas that missed the update will sync
> > with the leader when they recover.
>
>
> They have implemented this parameter called *min_rf* that you can use
> (client-side) to make sure that your update was replicated to at least one
> replica (e.g.: min_rf > 1).
>
> This is why my concern about moving storage around, because then I know
> when the shard leader comes back, solrcloud will run sync process for those
> documents that couldn't be sent to the replicas.
>
> Am I missing something or misunderstood the documentation ?
>
> Cheers !
>
>
>
>
>
>
>
> On 5 July 2016 at 19:49, Davis, Daniel (NIH/NLM) [C] <daniel.davis@nih.gov
> >
> wrote:
>
> > Lorenzo, this probably comes late, but my systems guys just don't want to
> > give me real disk. Although RAID-5 or LVM on-top of JBOD may be better
> > than Amazon EBS, Amazon EBS is still much closer to real disk in terms of
> > IOPS and latency than NFS ;) I even ran a mini test (not an official
> > benchmark), and found the response time for random reads to be better.
> >
> > If you are a young/smallish company, this may be all in the cloud, but if
> > you are in a large organization like mine, you may also need to allow for
> > other architectures, such as a "virtual" Netapp in the cloud that
> > communicates with a physical Netapp on-premises, and the
> throughput/latency
> > of that. The most important thing is to actually measure the numbers
> you
> > are getting, both for search and for simply raw I/O, or to get your
> > systems/storage guys to measure those numbers. If you get your
> > systems/storage guys to just measure storage - you will want to care
> about
> > three things for indexing primarily:
> >
> > Sequential Write Throughput
> > Random Read Throughput
> > Random Read Response Time/Latency
> >
> > Hope this helps,
> >
> > Dan Davis, Systems/Applications Architect (Contractor),
> > Office of Computer and Communications Systems,
> > National Library of Medicine, NIH
> >
> >
> >
> > -----Original Message-----
> > From: Lorenzo Fundaró [mailto:lorenzo.fundaro@dawandamail.com]
> > Sent: Tuesday, July 05, 2016 3:20 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: deploy solr on cloud providers
> >
> > Hi Shawn. Actually what im trying to find out is whether this is the best
> > approach for deploying solr in the cloud. I believe solrcloud solves a
> lot
> > of problems in terms of High Availability but when it comes to storage
> > there seems to be a limitation that can be workaround of course but it's
> a
> > bit cumbersome and i was wondering if there is a better option for this
> or
> > if im missing something with the way I'm doing it. I wonder if there are
> > some proved experience about how to solve the storage problem when
> > deploying in the cloud. Any advise or point to some enlightening
> > documentation will be appreciated. Thanks.
> > On Jul 4, 2016 18:27, "Shawn Heisey" <ap...@elyograg.org> wrote:
> >
> > > On 7/4/2016 10:18 AM, Lorenzo Fundaró wrote:
> > > > when deploying solr (in solrcloud mode) in the cloud one has to take
> > > > care of storage, and as far as I understand it can be a problem
> > > > because the storage should go wherever the node is created. If we
> > > > have for example, a node on EC2 with its own persistent disk, this
> > > > node happens to be the leader and at some point crashes but couldn't
> > > > make the replication of the data that has in the transaction log,
> > > > how do we do in that case ? Ideally the new node must use the
> > > > leftover data that the death node left, but this is a bit cumbersome
> > > > in my opinion. What are the best practices for this ?
> > >
> > > I can't make any sense of this. What is the *exact* problem you need
> > > to solve? The details can be very important.
> > >
> > > We might be dealing with this:
> > >
> > > http://people.apache.org/~hossman/#xyproblem
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
>
>
>
> --
>
> --
> Lorenzo Fundaro
> Backend Engineer
> E-Mail: lorenzo.fundaro@dawandamail.com
>
> Fax + 49 - (0)30 - 25 76 08 52
> Tel + 49 - (0)179 - 51 10 982
>
> DaWanda GmbH
> Windscheidstraße 18
> 10627 Berlin
>
> Geschäftsführer: Claudia Helming und Niels Nüssler
> AG Charlottenburg HRB 104695 B http://www.dawanda.com
>
Re: deploy solr on cloud providers
Posted by Lorenzo Fundaró <lo...@dawandamail.com>.
@Tomas and @Steven
I am a bit skeptical about this two statements:
If a node just disappears you should be fine in terms of data
> availability, since Solr in "SolrCloud" replicates the data as it comes it
> (before sending the http response)
and
>
> You shouldn't "need" to move the storage as SolrCloud will replicate all
> data to the new node and anything in the transaction log will already be
> distributed through the rest of the machines..
because according to the official documentation here
<https://cwiki.apache.org/confluence/display/solr/Read+and+Write+Side+Fault+Tolerance>:
(Write side fault tolerant -> recovery)
If a leader goes down, it may have sent requests to some replicas and not
> others. So when a new potential leader is identified, it runs a synch
> process against the other replicas. If this is successful, everything
> should be consistent, the leader registers as active, and normal actions
> proceed
I think there is a possibility that an update is not sent by the leader but
is kept in the local disk and after it comes up again it can sync the
non-sent data.
Furthermore:
Achieved Replication Factor
> When using a replication factor greater than one, an update request may
> succeed on the shard leader but fail on one or more of the replicas. For
> instance, consider a collection with one shard and replication factor of
> three. In this case, you have a shard leader and two additional replicas.
> If an update request succeeds on the leader but fails on both replicas, for
> whatever reason, the update request is still considered successful from the
> perspective of the client. The replicas that missed the update will sync
> with the leader when they recover.
They have implemented this parameter called *min_rf* that you can use
(client-side) to make sure that your update was replicated to at least one
replica (e.g.: min_rf > 1).
This is why my concern about moving storage around, because then I know
when the shard leader comes back, solrcloud will run sync process for those
documents that couldn't be sent to the replicas.
Am I missing something or misunderstood the documentation ?
Cheers !
On 5 July 2016 at 19:49, Davis, Daniel (NIH/NLM) [C] <da...@nih.gov>
wrote:
> Lorenzo, this probably comes late, but my systems guys just don't want to
> give me real disk. Although RAID-5 or LVM on-top of JBOD may be better
> than Amazon EBS, Amazon EBS is still much closer to real disk in terms of
> IOPS and latency than NFS ;) I even ran a mini test (not an official
> benchmark), and found the response time for random reads to be better.
>
> If you are a young/smallish company, this may be all in the cloud, but if
> you are in a large organization like mine, you may also need to allow for
> other architectures, such as a "virtual" Netapp in the cloud that
> communicates with a physical Netapp on-premises, and the throughput/latency
> of that. The most important thing is to actually measure the numbers you
> are getting, both for search and for simply raw I/O, or to get your
> systems/storage guys to measure those numbers. If you get your
> systems/storage guys to just measure storage - you will want to care about
> three things for indexing primarily:
>
> Sequential Write Throughput
> Random Read Throughput
> Random Read Response Time/Latency
>
> Hope this helps,
>
> Dan Davis, Systems/Applications Architect (Contractor),
> Office of Computer and Communications Systems,
> National Library of Medicine, NIH
>
>
>
> -----Original Message-----
> From: Lorenzo Fundaró [mailto:lorenzo.fundaro@dawandamail.com]
> Sent: Tuesday, July 05, 2016 3:20 AM
> To: solr-user@lucene.apache.org
> Subject: Re: deploy solr on cloud providers
>
> Hi Shawn. Actually what im trying to find out is whether this is the best
> approach for deploying solr in the cloud. I believe solrcloud solves a lot
> of problems in terms of High Availability but when it comes to storage
> there seems to be a limitation that can be workaround of course but it's a
> bit cumbersome and i was wondering if there is a better option for this or
> if im missing something with the way I'm doing it. I wonder if there are
> some proved experience about how to solve the storage problem when
> deploying in the cloud. Any advise or point to some enlightening
> documentation will be appreciated. Thanks.
> On Jul 4, 2016 18:27, "Shawn Heisey" <ap...@elyograg.org> wrote:
>
> > On 7/4/2016 10:18 AM, Lorenzo Fundaró wrote:
> > > when deploying solr (in solrcloud mode) in the cloud one has to take
> > > care of storage, and as far as I understand it can be a problem
> > > because the storage should go wherever the node is created. If we
> > > have for example, a node on EC2 with its own persistent disk, this
> > > node happens to be the leader and at some point crashes but couldn't
> > > make the replication of the data that has in the transaction log,
> > > how do we do in that case ? Ideally the new node must use the
> > > leftover data that the death node left, but this is a bit cumbersome
> > > in my opinion. What are the best practices for this ?
> >
> > I can't make any sense of this. What is the *exact* problem you need
> > to solve? The details can be very important.
> >
> > We might be dealing with this:
> >
> > http://people.apache.org/~hossman/#xyproblem
> >
> > Thanks,
> > Shawn
> >
> >
>
--
--
Lorenzo Fundaro
Backend Engineer
E-Mail: lorenzo.fundaro@dawandamail.com
Fax + 49 - (0)30 - 25 76 08 52
Tel + 49 - (0)179 - 51 10 982
DaWanda GmbH
Windscheidstraße 18
10627 Berlin
Geschäftsführer: Claudia Helming und Niels Nüssler
AG Charlottenburg HRB 104695 B http://www.dawanda.com
RE: deploy solr on cloud providers
Posted by "Davis, Daniel (NIH/NLM) [C]" <da...@nih.gov>.
Lorenzo, this probably comes late, but my systems guys just don't want to give me real disk. Although RAID-5 or LVM on-top of JBOD may be better than Amazon EBS, Amazon EBS is still much closer to real disk in terms of IOPS and latency than NFS ;) I even ran a mini test (not an official benchmark), and found the response time for random reads to be better.
If you are a young/smallish company, this may be all in the cloud, but if you are in a large organization like mine, you may also need to allow for other architectures, such as a "virtual" Netapp in the cloud that communicates with a physical Netapp on-premises, and the throughput/latency of that. The most important thing is to actually measure the numbers you are getting, both for search and for simply raw I/O, or to get your systems/storage guys to measure those numbers. If you get your systems/storage guys to just measure storage - you will want to care about three things for indexing primarily:
Sequential Write Throughput
Random Read Throughput
Random Read Response Time/Latency
Hope this helps,
Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH
-----Original Message-----
From: Lorenzo Fundaró [mailto:lorenzo.fundaro@dawandamail.com]
Sent: Tuesday, July 05, 2016 3:20 AM
To: solr-user@lucene.apache.org
Subject: Re: deploy solr on cloud providers
Hi Shawn. Actually what im trying to find out is whether this is the best approach for deploying solr in the cloud. I believe solrcloud solves a lot of problems in terms of High Availability but when it comes to storage there seems to be a limitation that can be workaround of course but it's a bit cumbersome and i was wondering if there is a better option for this or if im missing something with the way I'm doing it. I wonder if there are some proved experience about how to solve the storage problem when deploying in the cloud. Any advise or point to some enlightening documentation will be appreciated. Thanks.
On Jul 4, 2016 18:27, "Shawn Heisey" <ap...@elyograg.org> wrote:
> On 7/4/2016 10:18 AM, Lorenzo Fundaró wrote:
> > when deploying solr (in solrcloud mode) in the cloud one has to take
> > care of storage, and as far as I understand it can be a problem
> > because the storage should go wherever the node is created. If we
> > have for example, a node on EC2 with its own persistent disk, this
> > node happens to be the leader and at some point crashes but couldn't
> > make the replication of the data that has in the transaction log,
> > how do we do in that case ? Ideally the new node must use the
> > leftover data that the death node left, but this is a bit cumbersome
> > in my opinion. What are the best practices for this ?
>
> I can't make any sense of this. What is the *exact* problem you need
> to solve? The details can be very important.
>
> We might be dealing with this:
>
> http://people.apache.org/~hossman/#xyproblem
>
> Thanks,
> Shawn
>
>
Re: deploy solr on cloud providers
Posted by Lorenzo Fundaró <lo...@dawandamail.com>.
Hi Shawn. Actually what im trying to find out is whether this is the best
approach for deploying solr in the cloud. I believe solrcloud solves a lot
of problems in terms of High Availability but when it comes to storage
there seems to be a limitation that can be workaround of course but it's a
bit cumbersome and i was wondering if there is a better option for this or
if im missing something with the way I'm doing it. I wonder if there are
some proved experience about how to solve the storage problem when
deploying in the cloud. Any advise or point to some enlightening
documentation will be appreciated. Thanks.
On Jul 4, 2016 18:27, "Shawn Heisey" <ap...@elyograg.org> wrote:
> On 7/4/2016 10:18 AM, Lorenzo Fundaró wrote:
> > when deploying solr (in solrcloud mode) in the cloud one has to take
> > care of storage, and as far as I understand it can be a problem
> > because the storage should go wherever the node is created. If we have
> > for example, a node on EC2 with its own persistent disk, this node
> > happens to be the leader and at some point crashes but couldn't make
> > the replication of the data that has in the transaction log, how do we
> > do in that case ? Ideally the new node must use the leftover data that
> > the death node left, but this is a bit cumbersome in my opinion. What
> > are the best practices for this ?
>
> I can't make any sense of this. What is the *exact* problem you need to
> solve? The details can be very important.
>
> We might be dealing with this:
>
> http://people.apache.org/~hossman/#xyproblem
>
> Thanks,
> Shawn
>
>
Re: deploy solr on cloud providers
Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/4/2016 10:18 AM, Lorenzo Fundar� wrote:
> when deploying solr (in solrcloud mode) in the cloud one has to take
> care of storage, and as far as I understand it can be a problem
> because the storage should go wherever the node is created. If we have
> for example, a node on EC2 with its own persistent disk, this node
> happens to be the leader and at some point crashes but couldn't make
> the replication of the data that has in the transaction log, how do we
> do in that case ? Ideally the new node must use the leftover data that
> the death node left, but this is a bit cumbersome in my opinion. What
> are the best practices for this ?
I can't make any sense of this. What is the *exact* problem you need to
solve? The details can be very important.
We might be dealing with this:
http://people.apache.org/~hossman/#xyproblem
Thanks,
Shawn