Is Apache Ignite appropriate for my use case?

[steven <st...@gmail.com>]

Hi, I need to manage a large fleet of servers (around 100k machines). Each
node contains a subset of the data (basically binary blobs) in memory. The
data store is a separate database.

I am trying to provide a binary blob lookup service (in greatly simplified
terms). For each incoming request (which contains a binary string), every
node should perform matching of the string against its subset of the data.
When the first matching binary blob is found, that result is returned, and
all other nodes should stop searching. If no matches are found, NOT_FOUND
should be returned. Requests must be handled within 5 seconds.

If any node fails, it must be revived with the same subset of data that it
had before.

One challenge is how to handle node failure in the middle of a request. It
is unlikely that a node will be revived quickly enough to respond within 5
seconds. Most likely there should be standby nodes that will retrieve the
failing node's subset of data from the data store and perform matching upon
being notified of node failure.

Is Apache Ignite appropriate for this use case?

Thanks,
Steven



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is Apache Ignite appropriate for my use case?

[Denis Magda <dm...@apache.org>]

Folks,

If you have a replication factor set to 1 (aka. backup) then a loss of a
primary node won't affect the availability of your cluster as long as the
backup will be serving the requests.

-
Denis


On Wed, May 22, 2019 at 9:03 PM Coleman, JohnSteven (Agoda) <
JohnSteven.Coleman@agoda.com> wrote:

> Hi,
>
> Ignite can go so far in meeting your requirements for durable storage and
> retrieval, but a primary node failure is probably going to be terminal
> because unless you can boot a machine in less than 5 seconds you can forget
> about reloading ignite anyway, you simply won't be able to restore your TCP
> connection in time. (Unless of course you have some radical kind of
> hardware with OS in bubble memory or similar, but even then?)
>
> So you need to use something like a p2p protocol such as bit torrent where
> the download is broken into chunks and delegated to multiple nodes each
> holding some or all of the blobs data. The client has to be able to recover
> as well, either by round robin or by implementing a p2p protocol - you
> still can't depend on a single fallible point of entry to your service
> because of the problem above.
>
> If you have such a p2p protocol I'm not sure Ignite adds anything much of
> value?
>
> John
>
>
> -----Original Message-----
> From: steven <st...@gmail.com>
> Sent: Thursday, May 23, 2019 2:16 AM
> To: user@ignite.apache.org
> Subject: Is Apache Ignite appropriate for my use case?
>
> Email received from outside the company. If in doubt don't click links nor
> open attachments!
> ________________________________
>
> Hi, I need to manage a large fleet of servers (around 100k machines). Each
> node contains a subset of the data (basically binary blobs) in memory. The
> data store is a separate database.
>
> I am trying to provide a binary blob lookup service (in greatly simplified
> terms). For each incoming request (which contains a binary string), every
> node should perform matching of the string against its subset of the data.
> When the first matching binary blob is found, that result is returned, and
> all other nodes should stop searching. If no matches are found, NOT_FOUND
> should be returned. Requests must be handled within 5 seconds.
>
> If any node fails, it must be revived with the same subset of data that it
> had before.
>
> One challenge is how to handle node failure in the middle of a request. It
> is unlikely that a node will be revived quickly enough to respond within 5
> seconds. Most likely there should be standby nodes that will retrieve the
> failing node's subset of data from the data store and perform matching upon
> being notified of node failure.
>
> Is Apache Ignite appropriate for this use case?
>
> Thanks,
> Steven
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
> ________________________________
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by copyright
> or other legal rules. If you have received it by mistake please let us know
> by reply email and delete it from your system. It is prohibited to copy
> this message or disclose its content to anyone. Any confidentiality or
> privilege is not waived or lost by any mistaken delivery or unauthorized
> disclosure of the message. All messages sent to and from Agoda may be
> monitored to ensure compliance with company policies, to protect the
> company's interests and to remove potential malware. Electronic messages
> may be intercepted, amended, lost or deleted, or contain viruses.
>

RE: Is Apache Ignite appropriate for my use case?

["Coleman, JohnSteven (Agoda)" <Jo...@agoda.com>]

Hi,

Ignite can go so far in meeting your requirements for durable storage and retrieval, but a primary node failure is probably going to be terminal because unless you can boot a machine in less than 5 seconds you can forget about reloading ignite anyway, you simply won't be able to restore your TCP connection in time. (Unless of course you have some radical kind of hardware with OS in bubble memory or similar, but even then?)

So you need to use something like a p2p protocol such as bit torrent where the download is broken into chunks and delegated to multiple nodes each holding some or all of the blobs data. The client has to be able to recover as well, either by round robin or by implementing a p2p protocol - you still can't depend on a single fallible point of entry to your service because of the problem above.

If you have such a p2p protocol I'm not sure Ignite adds anything much of value?

John


-----Original Message-----
From: steven <st...@gmail.com>
Sent: Thursday, May 23, 2019 2:16 AM
To: user@ignite.apache.org
Subject: Is Apache Ignite appropriate for my use case?

Email received from outside the company. If in doubt don't click links nor open attachments!
________________________________

Hi, I need to manage a large fleet of servers (around 100k machines). Each node contains a subset of the data (basically binary blobs) in memory. The data store is a separate database.

I am trying to provide a binary blob lookup service (in greatly simplified terms). For each incoming request (which contains a binary string), every node should perform matching of the string against its subset of the data.
When the first matching binary blob is found, that result is returned, and all other nodes should stop searching. If no matches are found, NOT_FOUND should be returned. Requests must be handled within 5 seconds.

If any node fails, it must be revived with the same subset of data that it had before.

One challenge is how to handle node failure in the middle of a request. It is unlikely that a node will be revived quickly enough to respond within 5 seconds. Most likely there should be standby nodes that will retrieve the failing node's subset of data from the data store and perform matching upon being notified of node failure.

Is Apache Ignite appropriate for this use case?

Thanks,
Steven



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.