You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Osman YOZGATLIOĞLU <os...@kron.com.tr> on 2019/10/23 10:23:10 UTC
merge two cluster
Hello,
I have two cluster and both contains different data sets with different node counts.
Would it be beneficial to merge two cluster?
Regards,
Osman
Re: merge two cluster
Posted by Osman YOZGATLIOĞLU <os...@kron.com.tr>.
thank you all.
you saved my time and resource.
regards
osman
Get Outlook for Android<https://aka.ms/ghei36>
________________________________
From: Jon Haddad <jo...@jonhaddad.com>
Sent: Thursday, October 24, 2019 12:13:45 AM
To: user@cassandra.apache.org <us...@cassandra.apache.org>
Subject: Re: merge two cluster
Probably not beneficial, I wouldn't do it. Not a fan of multi-tenancy with Cassandra unless the use cases are so small that your noisy neighbor problem is not very noisy at all. For those cases I don't know what you get from Cassandra other than a cool resume.
On Wed, Oct 23, 2019 at 12:41 PM Reid Pinchback <rp...@tripadvisor.com>> wrote:
I haven’t seen much evidence that larger cluster = more performance, plus or minus the statistics of speculative retry. It horizontally scales for storage definitely, and somewhat for connection volume. If anything, per Sean’s observation, you have less ability to have a stable tuning for a particular usage pattern.
Try to have a mental picture of what you think is happening in the JVM while Cassandra is running. There are short-lived objects, medium-lived objects, long/static-lived objects, and behind the scenes some degree of read I/O and write I/O against disk. Garbage collectors struggle badly with medium-lived objects, but Cassandra really depends a great deal on those. If you merge two clusters together, within any one node you still have the JVM size and disk architecture you had before, but you are adding competition on fixed resources and potentially in the very way they find most difficult to handle.
If those resources were heavily underutilized, like Sean’s point about merging small apps together, then sure. But if those two clusters of yours are already showing that they experience significant load, then you are unlikely to improve anything, far more likely to end up worse off. GC overhead and compaction flushes to disk are your challenges; merging two clusters doesn’t change the physics of those two areas, but could increase the demand on them.
The only caveat to all of the above I can think of is if there was a fault-tolerance story motivating the merging. Like “management wants us in two AZs in AWS, but lacks the budget for more instances, and each pool by itself is too small for us to come up with a 2 rack organization that makes sense”.
R
From: Osman YOZGATLIOĞLU <os...@kron.com.tr>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Wednesday, October 23, 2019 at 10:40 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: merge two cluster
Message from External Sender
Sorry, missing question;
Actually I'm asking this for performance perspective. At application level both cluster used at the same time and approx same level. Inserted data inserted to both cluster, different parts of course.
If I merge two cluster, can I gain some performance improvements? Like raid stripes, more disk, more stripe, more speed..
Regards
On 23.10.2019 17:30, Durity, Sean R wrote:
Beneficial to whom? The apps, the admins, the developers?
I suggest that app teams have separate clusters per application. This prevents the noisy neighbor problem, isolates any security issues, and helps when it is time for maintenance, upgrade, performance testing, etc. to not have to coordinate multiple app teams at the same time. Also, an individual cluster can be tuned for its specific workload. Sometimes, though, costs and data size push us towards combining smaller apps owned by the same team onto a single cluster. Those are the exceptions.
As a Cassandra admin, I am always trying to scale the ability to admin multiple clusters without just adding new admins. That is an on-going task, dependent on your operating environment.
Also, because every table has a portion of memory (memtable), there is a practical limit to the number of tables that any one cluster should have. I have heard it is in the low hundreds of tables. This puts a limit on the number of applications that a cluster can safely support.
Sean Durity – Staff Systems Engineer, Cassandra
From: Osman YOZGATLIOĞLU <os...@kron.com.tr>
Sent: Wednesday, October 23, 2019 6:23 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: [EXTERNAL] merge two cluster
Hello,
I have two cluster and both contains different data sets with different node counts.
Would it be beneficial to merge two cluster?
Regards,
Osman
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Re: merge two cluster
Posted by Jon Haddad <jo...@jonhaddad.com>.
Probably not beneficial, I wouldn't do it. Not a fan of multi-tenancy with
Cassandra unless the use cases are so small that your noisy neighbor
problem is not very noisy at all. For those cases I don't know what you
get from Cassandra other than a cool resume.
On Wed, Oct 23, 2019 at 12:41 PM Reid Pinchback <rp...@tripadvisor.com>
wrote:
> I haven’t seen much evidence that larger cluster = more performance, plus
> or minus the statistics of speculative retry. It horizontally scales for
> storage definitely, and somewhat for connection volume. If anything, per
> Sean’s observation, you have less ability to have a stable tuning for a
> particular usage pattern.
>
>
>
> Try to have a mental picture of what you think is happening in the JVM
> while Cassandra is running. There are short-lived objects, medium-lived
> objects, long/static-lived objects, and behind the scenes some degree of
> read I/O and write I/O against disk. Garbage collectors struggle badly
> with medium-lived objects, but Cassandra really depends a great deal on
> those. If you merge two clusters together, within any one node you still
> have the JVM size and disk architecture you had before, but you are adding
> competition on fixed resources and potentially in the very way they find
> most difficult to handle.
>
>
>
> If those resources were heavily underutilized, like Sean’s point about
> merging small apps together, then sure. But if those two clusters of yours
> are already showing that they experience significant load, then you are
> unlikely to improve anything, far more likely to end up worse off. GC
> overhead and compaction flushes to disk are your challenges; merging two
> clusters doesn’t change the physics of those two areas, but could increase
> the demand on them.
>
>
>
> The only caveat to all of the above I can think of is if there was a
> fault-tolerance story motivating the merging. Like “management wants us in
> two AZs in AWS, but lacks the budget for more instances, and each pool by
> itself is too small for us to come up with a 2 rack organization that makes
> sense”.
>
>
>
> R
>
>
>
> *From: *Osman YOZGATLIOĞLU <os...@kron.com.tr>
> *Reply-To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
> *Date: *Wednesday, October 23, 2019 at 10:40 AM
> *To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
> *Subject: *Re: merge two cluster
>
>
>
> *Message from External Sender*
>
> Sorry, missing question;
>
> Actually I'm asking this for performance perspective. At application level
> both cluster used at the same time and approx same level. Inserted data
> inserted to both cluster, different parts of course.
>
> If I merge two cluster, can I gain some performance improvements? Like
> raid stripes, more disk, more stripe, more speed..
>
>
>
> Regards
>
> On 23.10.2019 17:30, Durity, Sean R wrote:
>
> Beneficial to whom? The apps, the admins, the developers?
>
>
>
> I suggest that app teams have separate clusters per application. This
> prevents the noisy neighbor problem, isolates any security issues, and
> helps when it is time for maintenance, upgrade, performance testing, etc.
> to not have to coordinate multiple app teams at the same time. Also, an
> individual cluster can be tuned for its specific workload. Sometimes,
> though, costs and data size push us towards combining smaller apps owned by
> the same team onto a single cluster. Those are the exceptions.
>
>
>
> As a Cassandra admin, I am always trying to scale the ability to admin
> multiple clusters without just adding new admins. That is an on-going task,
> dependent on your operating environment.
>
>
>
> Also, because every table has a portion of memory (memtable), there is a
> practical limit to the number of tables that any one cluster should have. I
> have heard it is in the low hundreds of tables. This puts a limit on the
> number of applications that a cluster can safely support.
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Osman YOZGATLIOĞLU <os...@kron.com.tr>
> <os...@kron.com.tr>
> *Sent:* Wednesday, October 23, 2019 6:23 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] merge two cluster
>
>
>
> Hello,
>
> I have two cluster and both contains different data sets with different
> node counts.
>
> Would it be beneficial to merge two cluster?
>
>
>
> Regards,
>
> Osman
>
>
> ------------------------------
>
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>
>
Re: merge two cluster
Posted by Reid Pinchback <rp...@tripadvisor.com>.
I haven’t seen much evidence that larger cluster = more performance, plus or minus the statistics of speculative retry. It horizontally scales for storage definitely, and somewhat for connection volume. If anything, per Sean’s observation, you have less ability to have a stable tuning for a particular usage pattern.
Try to have a mental picture of what you think is happening in the JVM while Cassandra is running. There are short-lived objects, medium-lived objects, long/static-lived objects, and behind the scenes some degree of read I/O and write I/O against disk. Garbage collectors struggle badly with medium-lived objects, but Cassandra really depends a great deal on those. If you merge two clusters together, within any one node you still have the JVM size and disk architecture you had before, but you are adding competition on fixed resources and potentially in the very way they find most difficult to handle.
If those resources were heavily underutilized, like Sean’s point about merging small apps together, then sure. But if those two clusters of yours are already showing that they experience significant load, then you are unlikely to improve anything, far more likely to end up worse off. GC overhead and compaction flushes to disk are your challenges; merging two clusters doesn’t change the physics of those two areas, but could increase the demand on them.
The only caveat to all of the above I can think of is if there was a fault-tolerance story motivating the merging. Like “management wants us in two AZs in AWS, but lacks the budget for more instances, and each pool by itself is too small for us to come up with a 2 rack organization that makes sense”.
R
From: Osman YOZGATLIOĞLU <os...@kron.com.tr>
Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Date: Wednesday, October 23, 2019 at 10:40 AM
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Subject: Re: merge two cluster
Message from External Sender
Sorry, missing question;
Actually I'm asking this for performance perspective. At application level both cluster used at the same time and approx same level. Inserted data inserted to both cluster, different parts of course.
If I merge two cluster, can I gain some performance improvements? Like raid stripes, more disk, more stripe, more speed..
Regards
On 23.10.2019 17:30, Durity, Sean R wrote:
Beneficial to whom? The apps, the admins, the developers?
I suggest that app teams have separate clusters per application. This prevents the noisy neighbor problem, isolates any security issues, and helps when it is time for maintenance, upgrade, performance testing, etc. to not have to coordinate multiple app teams at the same time. Also, an individual cluster can be tuned for its specific workload. Sometimes, though, costs and data size push us towards combining smaller apps owned by the same team onto a single cluster. Those are the exceptions.
As a Cassandra admin, I am always trying to scale the ability to admin multiple clusters without just adding new admins. That is an on-going task, dependent on your operating environment.
Also, because every table has a portion of memory (memtable), there is a practical limit to the number of tables that any one cluster should have. I have heard it is in the low hundreds of tables. This puts a limit on the number of applications that a cluster can safely support.
Sean Durity – Staff Systems Engineer, Cassandra
From: Osman YOZGATLIOĞLU <os...@kron.com.tr>
Sent: Wednesday, October 23, 2019 6:23 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: [EXTERNAL] merge two cluster
Hello,
I have two cluster and both contains different data sets with different node counts.
Would it be beneficial to merge two cluster?
Regards,
Osman
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Re: merge two cluster
Posted by Osman YOZGATLIOĞLU <os...@kron.com.tr>.
Sorry, missing question;
Actually I'm asking this for performance perspective. At application level both cluster used at the same time and approx same level. Inserted data inserted to both cluster, different parts of course.
If I merge two cluster, can I gain some performance improvements? Like raid stripes, more disk, more stripe, more speed..
Regards
On 23.10.2019 17:30, Durity, Sean R wrote:
Beneficial to whom? The apps, the admins, the developers?
I suggest that app teams have separate clusters per application. This prevents the noisy neighbor problem, isolates any security issues, and helps when it is time for maintenance, upgrade, performance testing, etc. to not have to coordinate multiple app teams at the same time. Also, an individual cluster can be tuned for its specific workload. Sometimes, though, costs and data size push us towards combining smaller apps owned by the same team onto a single cluster. Those are the exceptions.
As a Cassandra admin, I am always trying to scale the ability to admin multiple clusters without just adding new admins. That is an on-going task, dependent on your operating environment.
Also, because every table has a portion of memory (memtable), there is a practical limit to the number of tables that any one cluster should have. I have heard it is in the low hundreds of tables. This puts a limit on the number of applications that a cluster can safely support.
Sean Durity – Staff Systems Engineer, Cassandra
From: Osman YOZGATLIOĞLU <os...@kron.com.tr>
Sent: Wednesday, October 23, 2019 6:23 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: [EXTERNAL] merge two cluster
Hello,
I have two cluster and both contains different data sets with different node counts.
Would it be beneficial to merge two cluster?
Regards,
Osman
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
RE: merge two cluster
Posted by "Durity, Sean R" <SE...@homedepot.com>.
Beneficial to whom? The apps, the admins, the developers?
I suggest that app teams have separate clusters per application. This prevents the noisy neighbor problem, isolates any security issues, and helps when it is time for maintenance, upgrade, performance testing, etc. to not have to coordinate multiple app teams at the same time. Also, an individual cluster can be tuned for its specific workload. Sometimes, though, costs and data size push us towards combining smaller apps owned by the same team onto a single cluster. Those are the exceptions.
As a Cassandra admin, I am always trying to scale the ability to admin multiple clusters without just adding new admins. That is an on-going task, dependent on your operating environment.
Also, because every table has a portion of memory (memtable), there is a practical limit to the number of tables that any one cluster should have. I have heard it is in the low hundreds of tables. This puts a limit on the number of applications that a cluster can safely support.
Sean Durity – Staff Systems Engineer, Cassandra
From: Osman YOZGATLIOĞLU <os...@kron.com.tr>
Sent: Wednesday, October 23, 2019 6:23 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] merge two cluster
Hello,
I have two cluster and both contains different data sets with different node counts.
Would it be beneficial to merge two cluster?
Regards,
Osman
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.