You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Nima Movafaghrad <ni...@oracle.com> on 2014/06/05 02:07:27 UTC
Cluster Failure and Data Recovery
Hi everyone,
We are in the process of designing a high available system with zero data loss tolerance. Plan is for the spouts to read from a queue and process them down in several different specialized bolts and then flush to DB. How can we guarantee no data loss here? Should we keep the queue transactions open until data is committed to DB? Should we persist the state of all the bolts? What happens to the intermediate data if the whole cluster fails?
Any suggestions are much appreciated.
Nima
RE: Cluster Failure and Data Recovery
Posted by Nima Movafaghrad <ni...@oracle.com>.
Thanks Andrew. J
From: Andrew Montalenti [mailto:andrew@parsely.com]
Sent: Thursday, June 05, 2014 3:30 PM
To: user@storm.incubator.apache.org
Subject: Re: Cluster Failure and Data Recovery
Sounds like you might benefit from considering something like Kafka instead of a standard MQ. We have HYPERLINK "http://www.parsely.com/slides/logs/notes/#introducing-apache-kafka"some notes about this publicly online from our PyData talk on Kafka/Storm. You can configure Kafka to have an SLA on data that is in terms of data size or time; if your entire topology crashes or goes down, then you can resume messages at the spout from the moment the failure happened, and pay no penalty.
(Of course, then you need to figure out how to guarantee your Kafka plant is always online, but this is do-able given its distributed architecture.)
This doesn't sound like a problem that Storm should think about solving -- after all, if your entire Storm cluster fails, all of the high availability guarantees of each component are, by definition, out the window.
On Thu, Jun 5, 2014 at 2:08 PM, Nima Movafaghrad <HYPERLINK "mailto:nima.movafaghrad@oracle.com"nima.movafaghrad@oracle.com> wrote:
Thanks Srinath. We are already using the reliable message processing for bolts failure etc. My problem is with a catastrophic cases. For example, what happens if the entire cluster goes down or what if the Topology fully fails. At the moment we are reading from MQ and although keeping the transactions open would resolve our data loss prevention issue it isn't quiet feasible. Some of our bolts listen and batch for up to 30 seconds so they have big enough batches that can be committed to RDBMS. Keeping the transactions open for that long slows things down considerably.
So I guess to frame to question better I should ask, if there a way to persist the intermediate data?
Thanks,
Nima
From: Srinath C [mailto:HYPERLINK "mailto:srinath.c@gmail.com"srinath.c@gmail.com]
Sent: Wednesday, June 04, 2014 5:49 PM
To: user
Subject: Re: Cluster Failure and Data Recovery
Hi Nima,
Use the HYPERLINK "https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing"reliable message processing mechanism to ensure that there is no data loss. You would need support for transactional semantics from the tuple source where spout can commit/abort a read (kestrel, kafka, rabbitmq, etc can do this). Yes you would need to keep the queue transactions open until the spout receives an "ack" or "fail" for every tuple.
IMO, this ensures that each tuple is processed "atleast once" and not "exactly once" so you need to be prepared to end up with duplicate entries in your DB or have a way to figure out that a write to DB is duplicate or earlier write. This is case where there are crashes with intermediate data in memory.
Regards,
Srinath.
On Thu, Jun 5, 2014 at 5:37 AM, Nima Movafaghrad <HYPERLINK "mailto:nima.movafaghrad@oracle.com"nima.movafaghrad@oracle.com> wrote:
Hi everyone,
We are in the process of designing a high available system with zero data loss tolerance. Plan is for the spouts to read from a queue and process them down in several different specialized bolts and then flush to DB. How can we guarantee no data loss here? Should we keep the queue transactions open until data is committed to DB? Should we persist the state of all the bolts? What happens to the intermediate data if the whole cluster fails?
Any suggestions are much appreciated.
Nima
Re: Cluster Failure and Data Recovery
Posted by Andrew Montalenti <an...@parsely.com>.
Sounds like you might benefit from considering something like Kafka instead
of a standard MQ. We have some notes about this
<http://www.parsely.com/slides/logs/notes/#introducing-apache-kafka>
publicly online from our PyData talk on Kafka/Storm. You can configure
Kafka to have an SLA on data that is in terms of data size or time; if your
entire topology crashes or goes down, then you can resume messages at the
spout from the moment the failure happened, and pay no penalty.
(Of course, then you need to figure out how to guarantee your Kafka plant
is always online, but this is do-able given its distributed architecture.)
This doesn't sound like a problem that Storm should think about solving --
after all, if your entire Storm cluster fails, all of the high availability
guarantees of each component are, by definition, out the window.
On Thu, Jun 5, 2014 at 2:08 PM, Nima Movafaghrad <
nima.movafaghrad@oracle.com> wrote:
> Thanks Srinath. We are already using the reliable message processing for
> bolts failure etc. My problem is with a catastrophic cases. For example,
> what happens if the entire cluster goes down or what if the Topology fully
> fails. At the moment we are reading from MQ and although keeping the
> transactions open would resolve our data loss prevention issue it isn't
> quiet feasible. Some of our bolts listen and batch for up to 30 seconds so
> they have big enough batches that can be committed to RDBMS. Keeping the
> transactions open for that long slows things down considerably.
>
>
>
> So I guess to frame to question better I should ask, if there a way to
> persist the intermediate data?
>
>
>
> Thanks,
>
> Nima
>
>
>
> *From:* Srinath C [mailto:srinath.c@gmail.com]
> *Sent:* Wednesday, June 04, 2014 5:49 PM
> *To:* user
> *Subject:* Re: Cluster Failure and Data Recovery
>
>
>
> Hi Nima,
>
> Use the reliable message processing
> <https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing> mechanism
> to ensure that there is no data loss. You would need support for
> transactional semantics from the tuple source where spout can commit/abort
> a read (kestrel, kafka, rabbitmq, etc can do this). Yes you would need to
> keep the queue transactions open until the spout receives an "ack" or
> "fail" for every tuple.
>
> IMO, this ensures that each tuple is processed "atleast once" and not
> "exactly once" so you need to be prepared to end up with duplicate entries
> in your DB or have a way to figure out that a write to DB is duplicate or
> earlier write. This is case where there are crashes with intermediate data
> in memory.
>
>
>
> Regards,
>
> Srinath.
>
>
>
>
>
> On Thu, Jun 5, 2014 at 5:37 AM, Nima Movafaghrad <
> nima.movafaghrad@oracle.com> wrote:
>
> Hi everyone,
>
>
>
> We are in the process of designing a high available system with zero data
> loss tolerance. Plan is for the spouts to read from a queue and process
> them down in several different specialized bolts and then flush to DB. How
> can we guarantee no data loss here? Should we keep the queue transactions
> open until data is committed to DB? Should we persist the state of all the
> bolts? What happens to the intermediate data if the whole cluster fails?
>
>
>
> Any suggestions are much appreciated.
>
>
>
> Nima
>
>
>
RE: Cluster Failure and Data Recovery
Posted by Nima Movafaghrad <ni...@oracle.com>.
Thanks Srinath. We are already using the reliable message processing for bolts failure etc. My problem is with a catastrophic cases. For example, what happens if the entire cluster goes down or what if the Topology fully fails. At the moment we are reading from MQ and although keeping the transactions open would resolve our data loss prevention issue it isn’t quiet feasible. Some of our bolts listen and batch for up to 30 seconds so they have big enough batches that can be committed to RDBMS. Keeping the transactions open for that long slows things down considerably.
So I guess to frame to question better I should ask, if there a way to persist the intermediate data?
Thanks,
Nima
From: Srinath C [mailto:srinath.c@gmail.com]
Sent: Wednesday, June 04, 2014 5:49 PM
To: user
Subject: Re: Cluster Failure and Data Recovery
Hi Nima,
Use the HYPERLINK "https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing"reliable message processing mechanism to ensure that there is no data loss. You would need support for transactional semantics from the tuple source where spout can commit/abort a read (kestrel, kafka, rabbitmq, etc can do this). Yes you would need to keep the queue transactions open until the spout receives an "ack" or "fail" for every tuple.
IMO, this ensures that each tuple is processed "atleast once" and not "exactly once" so you need to be prepared to end up with duplicate entries in your DB or have a way to figure out that a write to DB is duplicate or earlier write. This is case where there are crashes with intermediate data in memory.
Regards,
Srinath.
On Thu, Jun 5, 2014 at 5:37 AM, Nima Movafaghrad <HYPERLINK "mailto:nima.movafaghrad@oracle.com"nima.movafaghrad@oracle.com> wrote:
Hi everyone,
We are in the process of designing a high available system with zero data loss tolerance. Plan is for the spouts to read from a queue and process them down in several different specialized bolts and then flush to DB. How can we guarantee no data loss here? Should we keep the queue transactions open until data is committed to DB? Should we persist the state of all the bolts? What happens to the intermediate data if the whole cluster fails?
Any suggestions are much appreciated.
Nima
Re: Cluster Failure and Data Recovery
Posted by Srinath C <sr...@gmail.com>.
Hi Nima,
Use the reliable message processing
<https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing>
mechanism
to ensure that there is no data loss. You would need support for
transactional semantics from the tuple source where spout can commit/abort
a read (kestrel, kafka, rabbitmq, etc can do this). Yes you would need to
keep the queue transactions open until the spout receives an "ack" or
"fail" for every tuple.
IMO, this ensures that each tuple is processed "atleast once" and not
"exactly once" so you need to be prepared to end up with duplicate entries
in your DB or have a way to figure out that a write to DB is duplicate or
earlier write. This is case where there are crashes with intermediate data
in memory.
Regards,
Srinath.
On Thu, Jun 5, 2014 at 5:37 AM, Nima Movafaghrad <
nima.movafaghrad@oracle.com> wrote:
> Hi everyone,
>
>
>
> We are in the process of designing a high available system with zero data
> loss tolerance. Plan is for the spouts to read from a queue and process
> them down in several different specialized bolts and then flush to DB. How
> can we guarantee no data loss here? Should we keep the queue transactions
> open until data is committed to DB? Should we persist the state of all the
> bolts? What happens to the intermediate data if the whole cluster fails?
>
>
>
> Any suggestions are much appreciated.
>
>
>
> Nima
>