You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Shiva Kumar <sh...@gmail.com> on 2019/08/21 13:16:57 UTC
Capacity planning for production deployment on kubernetes
Hi all,
we are testing field use case before deploying in the field and we want to
know whether below resource limits are suitable in production.
There are 3 nodes (3 pods on kubernetes) running. Each having below
configuration
DefaultDataRegion: 60GB
JVM: 32GB
Resource allocated for each container: 64GB
And ignite documents says (JVM+ All DataRegion) should not exceed 70% of
total RAM allocated to each node(container).
but we started testing with the above configuration and up to 9 days ignite
cluster was running successfully and there was some data ingestion but
suddenly pods crashed and they were unable to recover from the crash.
does the above resource configuration not good for node recovery??
Re: Capacity planning for production deployment on kubernetes
Posted by Denis Mekhanikov <dm...@gmail.com>.
Shiva,
What version of Ignite do you use?
Before version 2.7 Ignite used a different mechanism to limit the size of the WAL history. It used DataStorageConfiguration#walHistorySize property, which is currently deprecated. This is what’s explained on the internal documentation page.
In Ignite versions starting from 2.7 DataStorageConfiguration#maxWalArchiveSize property is used. It specifies the maximum size of WAL history in bytes.
As you said, it’s 4 times the checkpoint buffer size. Since you didn’t change the size of the checkpoint buffer, by default it’s 1 GB in your case (rules for its calculation can be found here: https://apacheignite.readme.io/docs/durable-memory-tuning#section-checkpointing-buffer-size)
So, maximum WAL history size is 4 GB.
Plus you need to add WAL size itself which is (64 MB per segment) x (10 segments) = 640 megabytes by default.
If you use Ignite 2.7 or newer, then 10 GB should be enough for WAL with archive. Failures should be investigated in this case.
But Ignite 2.6 and older use a different approach, so there is no strict limitation on the WAL archive size.
Denis
On 22 Aug 2019, 22:35 +0300, Denis Magda <dm...@apache.org>, wrote:
> Please share the whole log file. It might be the case that something goes wrong with volumes you attached to Ignite pods.
>
> -
> Denis
>
>
> > On Thu, Aug 22, 2019 at 8:07 AM Shiva Kumar <sh...@gmail.com> wrote:
> > > Hi Denis,
> > >
> > > Thanks for your response,
> > > yes in our test also we have seen OOM errors and pod crash.
> > > so we will follow the recommendation for RAM requirements and also I was checking to ignite documentation on disk space required for WAL + WAL archive.
> > > here in this link https://apacheignite.readme.io/docs/write-ahead-log#section-wal-archive
> > >
> > > it says: archive size is defined as 4 times the size of the checkpointing buffer and checkpointing buffer is a function of the data region (https://apacheignite.readme.io/docs/durable-memory-tuning#section-checkpointing-buffer-size)
> > >
> > > but in this link https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-SubfoldersGeneration
> > >
> > > under Estimating disk space section it explains something to estimate disk space required for WAL but it is not clear, can you please help me the correct recommendation for calculating the disk space required for WAL+WAL archive.
> > >
> > > In one of my testing, I configured 4GB for data region and 10GB for WAL+WAL archive but our pods crashing as disk mounted for WAL+WAL archive runs out of space.
> > >
> > > [ignite@ignite-cluster-ignite-node-2 ignite]$ df -h
> > > Filesystem Size Used Avail Use% Mounted on
> > > overlay 158G 39G 112G 26% /
> > > tmpfs 63G 0 63G 0% /dev
> > > tmpfs 63G 0 63G 0% /sys/fs/cgroup
> > > /dev/vda1 158G 39G 112G 26% /etc/hosts
> > > shm 64M 0 64M 0% /dev/shm
> > > /dev/vdq 9.8G 9.7G 44M 100% /opt/ignite/wal
> > > /dev/vdr 50G 1.4G 48G 3% /opt/ignite/persistence
> > > tmpfs 63G 12K 63G 1% /run/secrets/kubernetes.io/serviceaccount
> > > tmpfs 63G 0 63G 0% /proc/acpi
> > > tmpfs 63G 0 63G 0% /proc/scsi
> > > tmpfs 63G 0 63G 0% /sys/firmware
> > >
> > >
> > > and this is the error message in ignite node:
> > >
> > > "ERROR","JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteCheckedException: Failed to archive WAL segment [srcFile=/opt/ignite/wal/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0000000000000006.wal, dstFile=/opt/ignite/wal/archive/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0000000000000236.wal.tmp]]]"
> > >
> > > > On Thu, Aug 22, 2019 at 8:04 PM Denis Mekhanikov <dm...@gmail.com> wrote:
> > > > > Shivakumar,
> > > > >
> > > > > Such allocation doesn’t allow full memory utilization, so it’s possible, that nodes will crash because of out of memory errors.
> > > > > So, it’s better to follow the given recommendation.
> > > > >
> > > > > If you want us to investigate reasons of the failures, please provide logs and configuration of the failed nodes.
> > > > >
> > > > > Denis
> > > > > On 21 Aug 2019, 16:17 +0300, Shiva Kumar <sh...@gmail.com>, wrote:
> > > > > > Hi all,
> > > > > > we are testing field use case before deploying in the field and we want to know whether below resource limits are suitable in production.
> > > > > > There are 3 nodes (3 pods on kubernetes) running. Each having below configuration
> > > > > >
> > > > > > DefaultDataRegion: 60GB
> > > > > > JVM: 32GB
> > > > > > Resource allocated for each container: 64GB
> > > > > >
> > > > > > And ignite documents says (JVM+ All DataRegion) should not exceed 70% of total RAM allocated to each node(container).
> > > > > > but we started testing with the above configuration and up to 9 days ignite cluster was running successfully and there was some data ingestion but suddenly pods crashed and they were unable to recover from the crash.
> > > > > > does the above resource configuration not good for node recovery??
Re: Capacity planning for production deployment on kubernetes
Posted by Denis Magda <dm...@apache.org>.
Please share the whole log file. It might be the case that something goes
wrong with volumes you attached to Ignite pods.
-
Denis
On Thu, Aug 22, 2019 at 8:07 AM Shiva Kumar <sh...@gmail.com>
wrote:
> Hi Denis,
>
> Thanks for your response,
> yes in our test also we have seen OOM errors and pod crash.
> so we will follow the recommendation for RAM requirements and also I was
> checking to ignite documentation on disk space required for WAL + WAL
> archive.
> here in this link
> https://apacheignite.readme.io/docs/write-ahead-log#section-wal-archive
>
> it says: archive size is defined as 4 times the size of the checkpointing
> buffer and checkpointing buffer is a function of the data region (
> https://apacheignite.readme.io/docs/durable-memory-tuning#section-checkpointing-buffer-size
> )
>
> but in this link
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-SubfoldersGeneration
>
> under *Estimating disk space* section it explains something to estimate
> disk space required for WAL but it is not clear, can you please help me the
> correct recommendation for calculating the disk space required for WAL+WAL
> archive.
>
> In one of my testing, I configured 4GB for data region and 10GB for
> WAL+WAL archive but our pods crashing as disk mounted for WAL+WAL archive
> runs out of space.
>
> [ignite@ignite-cluster-ignite-node-2 ignite]$* df -h*
> Filesystem Size Used Avail Use% Mounted on
> overlay 158G 39G 112G 26% /
> tmpfs 63G 0 63G 0% /dev
> tmpfs 63G 0 63G 0% /sys/fs/cgroup
> /dev/vda1 158G 39G 112G 26% /etc/hosts
> shm 64M 0 64M 0% /dev/shm
> */dev/vdq 9.8G 9.7G 44M 100% /opt/ignite/wal*
> /dev/vdr 50G 1.4G 48G 3% /opt/ignite/persistence
> tmpfs 63G 12K 63G 1% /run/secrets/
> kubernetes.io/serviceaccount
> tmpfs 63G 0 63G 0% /proc/acpi
> tmpfs 63G 0 63G 0% /proc/scsi
> tmpfs 63G 0 63G 0% /sys/firmware
>
>
> and this is the error message in ignite node:
>
> "ERROR","JVM will be halted immediately due to the failure:
> [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
> o.a.i.IgniteCheckedException: Failed to archive WAL segment
> [srcFile=/opt/ignite/wal/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0000000000000006.wal,
> dstFile=/opt/ignite/wal/archive/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0000000000000236.wal.tmp]]]"
>
>
> On Thu, Aug 22, 2019 at 8:04 PM Denis Mekhanikov <dm...@gmail.com>
> wrote:
>
>> Shivakumar,
>>
>> Such allocation doesn’t allow full memory utilization, so it’s possible,
>> that nodes will crash because of out of memory errors.
>> So, it’s better to follow the given recommendation.
>>
>> If you want us to investigate reasons of the failures, please provide
>> logs and configuration of the failed nodes.
>>
>> Denis
>> On 21 Aug 2019, 16:17 +0300, Shiva Kumar <sh...@gmail.com>,
>> wrote:
>>
>> Hi all,
>> we are testing field use case before deploying in the field and we want
>> to know whether below resource limits are suitable in production.
>> There are 3 nodes (3 pods on kubernetes) running. Each having below
>> configuration
>>
>> DefaultDataRegion: 60GB
>> JVM: 32GB
>> Resource allocated for each container: 64GB
>>
>> And ignite documents says (JVM+ All DataRegion) should not exceed 70% of
>> total RAM allocated to each node(container).
>> but we started testing with the above configuration and up to 9 days
>> ignite cluster was running successfully and there was some data ingestion
>> but suddenly pods crashed and they were unable to recover from the crash.
>> does the above resource configuration not good for node recovery??
>>
>>
Re: Capacity planning for production deployment on kubernetes
Posted by Shiva Kumar <sh...@gmail.com>.
Hi Denis,
Thanks for your response,
yes in our test also we have seen OOM errors and pod crash.
so we will follow the recommendation for RAM requirements and also I was
checking to ignite documentation on disk space required for WAL + WAL
archive.
here in this link
https://apacheignite.readme.io/docs/write-ahead-log#section-wal-archive
it says: archive size is defined as 4 times the size of the checkpointing
buffer and checkpointing buffer is a function of the data region (
https://apacheignite.readme.io/docs/durable-memory-tuning#section-checkpointing-buffer-size
)
but in this link
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-SubfoldersGeneration
under *Estimating disk space* section it explains something to estimate
disk space required for WAL but it is not clear, can you please help me the
correct recommendation for calculating the disk space required for WAL+WAL
archive.
In one of my testing, I configured 4GB for data region and 10GB for WAL+WAL
archive but our pods crashing as disk mounted for WAL+WAL archive runs out
of space.
[ignite@ignite-cluster-ignite-node-2 ignite]$* df -h*
Filesystem Size Used Avail Use% Mounted on
overlay 158G 39G 112G 26% /
tmpfs 63G 0 63G 0% /dev
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/vda1 158G 39G 112G 26% /etc/hosts
shm 64M 0 64M 0% /dev/shm
*/dev/vdq 9.8G 9.7G 44M 100% /opt/ignite/wal*
/dev/vdr 50G 1.4G 48G 3% /opt/ignite/persistence
tmpfs 63G 12K 63G 1% /run/secrets/
kubernetes.io/serviceaccount
tmpfs 63G 0 63G 0% /proc/acpi
tmpfs 63G 0 63G 0% /proc/scsi
tmpfs 63G 0 63G 0% /sys/firmware
and this is the error message in ignite node:
"ERROR","JVM will be halted immediately due to the failure:
[failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
o.a.i.IgniteCheckedException: Failed to archive WAL segment
[srcFile=/opt/ignite/wal/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0000000000000006.wal,
dstFile=/opt/ignite/wal/archive/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0000000000000236.wal.tmp]]]"
On Thu, Aug 22, 2019 at 8:04 PM Denis Mekhanikov <dm...@gmail.com>
wrote:
> Shivakumar,
>
> Such allocation doesn’t allow full memory utilization, so it’s possible,
> that nodes will crash because of out of memory errors.
> So, it’s better to follow the given recommendation.
>
> If you want us to investigate reasons of the failures, please provide logs
> and configuration of the failed nodes.
>
> Denis
> On 21 Aug 2019, 16:17 +0300, Shiva Kumar <sh...@gmail.com>,
> wrote:
>
> Hi all,
> we are testing field use case before deploying in the field and we want to
> know whether below resource limits are suitable in production.
> There are 3 nodes (3 pods on kubernetes) running. Each having below
> configuration
>
> DefaultDataRegion: 60GB
> JVM: 32GB
> Resource allocated for each container: 64GB
>
> And ignite documents says (JVM+ All DataRegion) should not exceed 70% of
> total RAM allocated to each node(container).
> but we started testing with the above configuration and up to 9 days
> ignite cluster was running successfully and there was some data ingestion
> but suddenly pods crashed and they were unable to recover from the crash.
> does the above resource configuration not good for node recovery??
>
>
Re: Capacity planning for production deployment on kubernetes
Posted by Denis Mekhanikov <dm...@gmail.com>.
Shivakumar,
Such allocation doesn’t allow full memory utilization, so it’s possible, that nodes will crash because of out of memory errors.
So, it’s better to follow the given recommendation.
If you want us to investigate reasons of the failures, please provide logs and configuration of the failed nodes.
Denis
On 21 Aug 2019, 16:17 +0300, Shiva Kumar <sh...@gmail.com>, wrote:
> Hi all,
> we are testing field use case before deploying in the field and we want to know whether below resource limits are suitable in production.
> There are 3 nodes (3 pods on kubernetes) running. Each having below configuration
>
> DefaultDataRegion: 60GB
> JVM: 32GB
> Resource allocated for each container: 64GB
>
> And ignite documents says (JVM+ All DataRegion) should not exceed 70% of total RAM allocated to each node(container).
> but we started testing with the above configuration and up to 9 days ignite cluster was running successfully and there was some data ingestion but suddenly pods crashed and they were unable to recover from the crash.
> does the above resource configuration not good for node recovery??