You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@ignite.apache.org by Amit Pundir <am...@gmail.com> on 2017/07/18 23:22:43 UTC

Multi Cloud Deployment Issue

Hi,
I am trying to setup a multi-cloud cluster of Ignite 2.0 servers and facing
severe issues in terms of the response time. I am running Tomcats as Ignite
clients hosting my REST service.

To narrow down the issue, I tried setting up the cluster in almost all
permutation combinations.

1. Two Ignite server nodes each on two clouds Cloud 1 and Cloud 2. Similarly
two Tomcats are set up on the same 2 clouds each.
2. Two Ignite server nodes each on two clouds Cloud 1 and Cloud 2. But all 4
Tomcats on Cloud 1.
3. Two Ignite server nodes each on two clouds Cloud 1 and Cloud 2. But all 4
Tomcats on Cloud 2.
4. Four Ignite server nodes each on Cloud 1. But 4 Tomcats split equally on
2 clouds.
5. Four Ignite server nodes each on Cloud 1. All 4 Tomcats on same cloud 1.
6. Four Ignite server nodes each on Cloud 2. All 4 Tomcats on same cloud 2.

As expected the case 5 and case 6 were the winners hands down but I can't
afford to deploy my application on a single cloud.

The average response time in case 1 and case 2 is 10 times more than what I
am getting on same cloud. I tried this experiment with 2 clouds hosted in
the same data-center and it was still 8 to 9 times bad than the best case
scenario.

The round trip time between these 2 clouds is 40 milli-secs and I was
expecting deterioration in the responsive time proportional to the round
trip time but that's not the case.

I observed the slowest api calls taking around 5 to 12 seconds. The 2
prominent calls were -

1. Ignite transaction commit()
2. TcpCommunicationSpi#createTcpClient()

Could you please help me on this issue.

While trying things out, I have increased the idleTimeout to 60 seconds to
reduce the calls to create new connections. What are the pitfalls of
increasing this timeout?

Thanks

--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Multi-Cloud-Deployment-Issue-tp15092.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Multi Cloud Deployment Issue

Posted by Amit Pundir <am...@gmail.com>.

Hi Mikhail,
It is critical for us to prove Ignite performance on atleast 2 clouds (not
necessarily data center replication) before enterprise wide adoption. Any
tips to improve its performance would be appreciated.

Thanks



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Multi-Cloud-Deployment-Issue-tp15092p15203.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Multi Cloud Deployment Issue

Posted by Amit Pundir <am...@gmail.com>.

HI Mikhail,
Thanks for the response.

I am aware that the data center replication/HA is available in the paid
version of Ignite. At this point, I don't have the option to go for it.

What I am trying instead is to minimize my risk by distributing the data
between different clouds not necessarily guaranteeing HA. The 2 clouds where
I am hosting this cluster have a latency of 10 milli-seconds only on a very
robust network.

In the cluster config file, all I am mentioning is the IP addresses. This
kind of latency can also be present in the same cloud but why the
performance degrades so much when the IPs belong to different clouds.

Does Ignite internally check if the IPs belong to the same network or not? 
I am looking for options to alleviate the risks associated with single
cloud. Any help is appreciated.


Thanks





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Multi-Cloud-Deployment-Issue-tp15092p15135.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Multi Cloud Deployment Issue

Posted by Amit Pundir <am...@gmail.com>.

Hi Mikhail, 
It is critical for us to prove Ignite performance on atleast 2 clouds (not
necessarily data center replication) before enterprise wide adoption. Any
tips to improve its performance would be appreciated. 

Thanks



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Multi-Cloud-Deployment-Issue-tp15092p15208.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Multi Cloud Deployment Issue

Posted by Mikhail Cherkasov <mc...@gridgain.com>.

Hi Amit,

it's expected penalty, ignite is cluster solution that requires fast and
reliable network.

Do you need share data between data centers for HA? if so, you can
implement
your own data center replication: install listener for cache changes, batch
data and send them
to remote data center, or you can try to google 3dparty solutions for this.

Thanks,
Mikhail.

On Wed, Jul 19, 2017 at 2:22 AM, Amit Pundir <am...@gmail.com> wrote:

> Hi,
> I am trying to setup a multi-cloud cluster of Ignite 2.0 servers and facing
> severe issues in terms of the response time. I am running Tomcats as Ignite
> clients hosting my REST service.
>
> To narrow down the issue, I tried setting up the cluster in almost all
> permutation combinations.
>
> 1. Two Ignite server nodes each on two clouds Cloud 1 and Cloud 2.
> Similarly
> two Tomcats are set up on the same 2 clouds each.
> 2. Two Ignite server nodes each on two clouds Cloud 1 and Cloud 2. But all
> 4
> Tomcats on Cloud 1.
> 3. Two Ignite server nodes each on two clouds Cloud 1 and Cloud 2. But all
> 4
> Tomcats on Cloud 2.
> 4. Four Ignite server nodes each on Cloud 1. But 4 Tomcats split equally on
> 2 clouds.
> 5. Four Ignite server nodes each on Cloud 1. All 4 Tomcats on same cloud 1.
> 6. Four Ignite server nodes each on Cloud 2. All 4 Tomcats on same cloud 2.
>
> As expected the case 5 and case 6 were the winners hands down but I can't
> afford to deploy my application on a single cloud.
>
> The average response time in case 1 and case 2 is 10 times more than what I
> am getting on same cloud. I tried this experiment with 2 clouds hosted in
> the same data-center and it was still 8 to 9 times bad than the best case
> scenario.
>
> The round trip time between these 2 clouds is 40 milli-secs and I was
> expecting deterioration in the responsive time proportional to the round
> trip time but that's not the case.
>
> I observed the slowest api calls taking around 5 to 12 seconds. The 2
> prominent calls were -
>
> 1. Ignite transaction commit()
> 2. TcpCommunicationSpi#createTcpClient()
>
> Could you please help me on this issue.
>
> While trying things out, I have increased the idleTimeout to 60 seconds to
> reduce the calls to create new connections. What are the pitfalls of
> increasing this timeout?
>
> Thanks
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Multi-Cloud-Deployment-Issue-tp15092.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Thanks,
Mikhail.