You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@fineract.apache.org by Karthik Ananthakrishnan <ka...@gmail.com> on 2018/10/01 05:23:48 UTC
Re: Fineract CN Server - Insufficient memory - Java Run time error
Hey Victor,
I did check for different metrics to analyse server performance in
cloudwatch before trying out swap space. Nginx, MySQL, Cassandra and
Fineract Application are the only resource consuming applications. Each
microservice takes around 2GB of memory when starting.. so, 12 micro
services taking up around 25GB of Memory space leaving very limited
resource for MySQL, Cassandra, Nginx to run. For now i am not considering
elastic scalability. This is the reason, i commented out few microservices
and starting the server to optimise memory consumption.
Regards,
Karthik
On Mon, Oct 1, 2018 at 1:42 AM Victor Manuel Romero Rodriguez <
vic_romero@hotmail.com> wrote:
> Hi Karthik,
>
> You have faced the error that all might face when running the services.
>
> I can view in the image log that you are using a EC2 machine, you can
> check the behaviour of the server and make the proper changes using
> CloudWatch.
>
> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring_ec2.html
>
> Changes are not only at JVM, but also at OS level and of course the
> Elastic Cloud could help to jump from one machine size to another in a
> "smooth" way (always there is pain).
>
> Even for testing/demo you should check what is the resource consumption
> for suggesting or proposing a better sizing of the machine based on the
> services that will be enabled.
>
> Regards
>
> Victor
>
>
> El 30/09/18 a las 14:06, Karthik Ananthakrishnan escribió:
> > Hello Isaac and others,
> >
> > Many thanks for the suggestions. I tried with xms4G and was not very
> > helpful. When i looked at the root cause, I found that cassandra trips
> > during the server boot process when the memory hits peak(free memory less
> > than 100MB) in 32GB RAM server. So, i added swap space to it which
> helped.
> > But still the cassandra would go down when memory hits peak. So, i
> decided
> > to run the server with fewer microservices by commenting out a few. Once
> i
> > did that server started properly and has been stable for around a week.
> > Even in local machine, i could run the application by running fewer
> > microservices.
> >
> > Thanks,
> > Karthik
> >
> > On Tue, Sep 18, 2018 at 10:31 PM Karthik Ananthakrishnan <
> > karthik.a@beyondsquare.com> wrote:
> >
> >> Hey Issac. I did try the 2G and didnt really help. So, i have already
> made
> >> changes as xms2G and Xmx4G. Will get back to you if it worked.
> >>
> >> On Tue, Sep 18, 2018 at 7:20 PM, Isaac Kamga <is...@mifos.org>
> >> wrote:
> >>
> >>> Thanks for helping out with the image Ed.
> >>>
> >>> The image never showed up for me...probably a pesky internet
> connection.
> >>>
> >>> @Karthik Ananthakrishnan <ka...@gmail.com> , I think my
> >>> aforementioned
> >>> suggestion should work. If it doesn't, then use `* -Xms4G -Xmx4G* `
> >>> instead. Also, provide feedback on whether or not it helped and update
> >>> documentation appropriately.
> >>>
> >>> Cheers,
> >>> Isaac Kamga
> >>>
> >>> On Tue, Sep 18, 2018 at 2:43 PM Ed Cable <ed...@mifos.org> wrote:
> >>>
> >>>> The image showed up for me. I've added it to imgur here:
> >>>>
> >>>> https://imgur.com/a/Qt145wA
> >>>>
> >>>> Ed
> >>>>
> >>>> On Tue, Sep 18, 2018 at 3:50 AM Courage Angeh <courageangeh@gmail.com
> >
> >>>> wrote:
> >>>>
> >>>>> Hi Karthik,
> >>>>>
> >>>>> You still haven't attached the image. You could try using
> >>>>> https://pastebin.com/ instead.
> >>>>>
> >>>>> On Mon, Sep 17, 2018 at 11:54 AM Karthik Ananthakrishnan
> >>>>> <ka...@beyondsquare.com.invalid> wrote:
> >>>>>
> >>>>>> Thanks Issac. Attached the image as well as the err log for your
> >>>>>> reference. Let me try using the xms and xmx options. Are you using
> >>>> those
> >>>>>> when you run the server.?
> >>>>>>
> >>>>>>
> >>>>
> >>>> --
> >>>> *Ed Cable*
> >>>> President/CEO, Mifos Initiative
> >>>> edcable@mifos.org | Skype: edcable | Mobile: +1.484.477.8649
> >>>>
> >>>> *Collectively Creating a World of 3 Billion Maries | *
> http://mifos.org
> >>>> <http://facebook.com/mifos> <http://www.twitter.com/mifos>
> >>>>
> >>
>
>
--
Karthik Krishnan
Re: Fineract CN Server - Insufficient memory - Java Run time error
Posted by Karthik Ananthakrishnan <ka...@beyondsquare.com.INVALID>.
It is possible. But for now. I have only one EC2 machine provisioned. If i
need to enable more micro services, my plan is to run Cassandra and MySQL
is RDS and connect it from the application.
On Mon, Oct 1, 2018 at 11:47 AM Awasum Yannick <ya...@gmail.com>
wrote:
> Is it possible to run Cassandra and MySQL in a different VM? This might
> reduce the load on the single VM. Just have to connect the data stores to
> the other services.
>
>
> Thanks.
> Awasum
>
> On Mon, Oct 1, 2018, 06:24 Karthik Ananthakrishnan <ka...@gmail.com>
> wrote:
>
>> Hey Victor,
>>
>> I did check for different metrics to analyse server performance in
>> cloudwatch before trying out swap space. Nginx, MySQL, Cassandra and
>> Fineract Application are the only resource consuming applications. Each
>> microservice takes around 2GB of memory when starting.. so, 12 micro
>> services taking up around 25GB of Memory space leaving very limited
>> resource for MySQL, Cassandra, Nginx to run. For now i am not considering
>> elastic scalability. This is the reason, i commented out few microservices
>> and starting the server to optimise memory consumption.
>>
>> Regards,
>> Karthik
>>
>> On Mon, Oct 1, 2018 at 1:42 AM Victor Manuel Romero Rodriguez <
>> vic_romero@hotmail.com> wrote:
>>
>> > Hi Karthik,
>> >
>> > You have faced the error that all might face when running the services.
>> >
>> > I can view in the image log that you are using a EC2 machine, you can
>> > check the behaviour of the server and make the proper changes using
>> > CloudWatch.
>> >
>> > https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring_ec2.html
>> >
>> > Changes are not only at JVM, but also at OS level and of course the
>> > Elastic Cloud could help to jump from one machine size to another in a
>> > "smooth" way (always there is pain).
>> >
>> > Even for testing/demo you should check what is the resource consumption
>> > for suggesting or proposing a better sizing of the machine based on the
>> > services that will be enabled.
>> >
>> > Regards
>> >
>> > Victor
>> >
>> >
>> > El 30/09/18 a las 14:06, Karthik Ananthakrishnan escribió:
>> > > Hello Isaac and others,
>> > >
>> > > Many thanks for the suggestions. I tried with xms4G and was not very
>> > > helpful. When i looked at the root cause, I found that cassandra trips
>> > > during the server boot process when the memory hits peak(free memory
>> less
>> > > than 100MB) in 32GB RAM server. So, i added swap space to it which
>> > helped.
>> > > But still the cassandra would go down when memory hits peak. So, i
>> > decided
>> > > to run the server with fewer microservices by commenting out a few.
>> Once
>> > i
>> > > did that server started properly and has been stable for around a
>> week.
>> > > Even in local machine, i could run the application by running fewer
>> > > microservices.
>> > >
>> > > Thanks,
>> > > Karthik
>> > >
>> > > On Tue, Sep 18, 2018 at 10:31 PM Karthik Ananthakrishnan <
>> > > karthik.a@beyondsquare.com> wrote:
>> > >
>> > >> Hey Issac. I did try the 2G and didnt really help. So, i have already
>> > made
>> > >> changes as xms2G and Xmx4G. Will get back to you if it worked.
>> > >>
>> > >> On Tue, Sep 18, 2018 at 7:20 PM, Isaac Kamga <is...@mifos.org>
>> > >> wrote:
>> > >>
>> > >>> Thanks for helping out with the image Ed.
>> > >>>
>> > >>> The image never showed up for me...probably a pesky internet
>> > connection.
>> > >>>
>> > >>> @Karthik Ananthakrishnan <ka...@gmail.com> , I think my
>> > >>> aforementioned
>> > >>> suggestion should work. If it doesn't, then use `* -Xms4G -Xmx4G* `
>> > >>> instead. Also, provide feedback on whether or not it helped and
>> update
>> > >>> documentation appropriately.
>> > >>>
>> > >>> Cheers,
>> > >>> Isaac Kamga
>> > >>>
>> > >>> On Tue, Sep 18, 2018 at 2:43 PM Ed Cable <ed...@mifos.org> wrote:
>> > >>>
>> > >>>> The image showed up for me. I've added it to imgur here:
>> > >>>>
>> > >>>> https://imgur.com/a/Qt145wA
>> > >>>>
>> > >>>> Ed
>> > >>>>
>> > >>>> On Tue, Sep 18, 2018 at 3:50 AM Courage Angeh <
>> courageangeh@gmail.com
>> > >
>> > >>>> wrote:
>> > >>>>
>> > >>>>> Hi Karthik,
>> > >>>>>
>> > >>>>> You still haven't attached the image. You could try using
>> > >>>>> https://pastebin.com/ instead.
>> > >>>>>
>> > >>>>> On Mon, Sep 17, 2018 at 11:54 AM Karthik Ananthakrishnan
>> > >>>>> <ka...@beyondsquare.com.invalid> wrote:
>> > >>>>>
>> > >>>>>> Thanks Issac. Attached the image as well as the err log for your
>> > >>>>>> reference. Let me try using the xms and xmx options. Are you
>> using
>> > >>>> those
>> > >>>>>> when you run the server.?
>> > >>>>>>
>> > >>>>>>
>> > >>>>
>> > >>>> --
>> > >>>> *Ed Cable*
>> > >>>> President/CEO, Mifos Initiative
>> > >>>> edcable@mifos.org | Skype: edcable | Mobile: +1.484.477.8649
>> > >>>>
>> > >>>> *Collectively Creating a World of 3 Billion Maries | *
>> > http://mifos.org
>> > >>>> <http://facebook.com/mifos> <http://www.twitter.com/mifos>
>> > >>>>
>> > >>
>> >
>> >
>>
>> --
>> Karthik Krishnan
>>
>
Re: Fineract CN Server - Insufficient memory - Java Run time error
Posted by Awasum Yannick <ya...@gmail.com>.
Is it possible to run Cassandra and MySQL in a different VM? This might
reduce the load on the single VM. Just have to connect the data stores to
the other services.
Thanks.
Awasum
On Mon, Oct 1, 2018, 06:24 Karthik Ananthakrishnan <ka...@gmail.com>
wrote:
> Hey Victor,
>
> I did check for different metrics to analyse server performance in
> cloudwatch before trying out swap space. Nginx, MySQL, Cassandra and
> Fineract Application are the only resource consuming applications. Each
> microservice takes around 2GB of memory when starting.. so, 12 micro
> services taking up around 25GB of Memory space leaving very limited
> resource for MySQL, Cassandra, Nginx to run. For now i am not considering
> elastic scalability. This is the reason, i commented out few microservices
> and starting the server to optimise memory consumption.
>
> Regards,
> Karthik
>
> On Mon, Oct 1, 2018 at 1:42 AM Victor Manuel Romero Rodriguez <
> vic_romero@hotmail.com> wrote:
>
> > Hi Karthik,
> >
> > You have faced the error that all might face when running the services.
> >
> > I can view in the image log that you are using a EC2 machine, you can
> > check the behaviour of the server and make the proper changes using
> > CloudWatch.
> >
> > https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring_ec2.html
> >
> > Changes are not only at JVM, but also at OS level and of course the
> > Elastic Cloud could help to jump from one machine size to another in a
> > "smooth" way (always there is pain).
> >
> > Even for testing/demo you should check what is the resource consumption
> > for suggesting or proposing a better sizing of the machine based on the
> > services that will be enabled.
> >
> > Regards
> >
> > Victor
> >
> >
> > El 30/09/18 a las 14:06, Karthik Ananthakrishnan escribió:
> > > Hello Isaac and others,
> > >
> > > Many thanks for the suggestions. I tried with xms4G and was not very
> > > helpful. When i looked at the root cause, I found that cassandra trips
> > > during the server boot process when the memory hits peak(free memory
> less
> > > than 100MB) in 32GB RAM server. So, i added swap space to it which
> > helped.
> > > But still the cassandra would go down when memory hits peak. So, i
> > decided
> > > to run the server with fewer microservices by commenting out a few.
> Once
> > i
> > > did that server started properly and has been stable for around a week.
> > > Even in local machine, i could run the application by running fewer
> > > microservices.
> > >
> > > Thanks,
> > > Karthik
> > >
> > > On Tue, Sep 18, 2018 at 10:31 PM Karthik Ananthakrishnan <
> > > karthik.a@beyondsquare.com> wrote:
> > >
> > >> Hey Issac. I did try the 2G and didnt really help. So, i have already
> > made
> > >> changes as xms2G and Xmx4G. Will get back to you if it worked.
> > >>
> > >> On Tue, Sep 18, 2018 at 7:20 PM, Isaac Kamga <is...@mifos.org>
> > >> wrote:
> > >>
> > >>> Thanks for helping out with the image Ed.
> > >>>
> > >>> The image never showed up for me...probably a pesky internet
> > connection.
> > >>>
> > >>> @Karthik Ananthakrishnan <ka...@gmail.com> , I think my
> > >>> aforementioned
> > >>> suggestion should work. If it doesn't, then use `* -Xms4G -Xmx4G* `
> > >>> instead. Also, provide feedback on whether or not it helped and
> update
> > >>> documentation appropriately.
> > >>>
> > >>> Cheers,
> > >>> Isaac Kamga
> > >>>
> > >>> On Tue, Sep 18, 2018 at 2:43 PM Ed Cable <ed...@mifos.org> wrote:
> > >>>
> > >>>> The image showed up for me. I've added it to imgur here:
> > >>>>
> > >>>> https://imgur.com/a/Qt145wA
> > >>>>
> > >>>> Ed
> > >>>>
> > >>>> On Tue, Sep 18, 2018 at 3:50 AM Courage Angeh <
> courageangeh@gmail.com
> > >
> > >>>> wrote:
> > >>>>
> > >>>>> Hi Karthik,
> > >>>>>
> > >>>>> You still haven't attached the image. You could try using
> > >>>>> https://pastebin.com/ instead.
> > >>>>>
> > >>>>> On Mon, Sep 17, 2018 at 11:54 AM Karthik Ananthakrishnan
> > >>>>> <ka...@beyondsquare.com.invalid> wrote:
> > >>>>>
> > >>>>>> Thanks Issac. Attached the image as well as the err log for your
> > >>>>>> reference. Let me try using the xms and xmx options. Are you
> using
> > >>>> those
> > >>>>>> when you run the server.?
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>> --
> > >>>> *Ed Cable*
> > >>>> President/CEO, Mifos Initiative
> > >>>> edcable@mifos.org | Skype: edcable | Mobile: +1.484.477.8649
> > >>>>
> > >>>> *Collectively Creating a World of 3 Billion Maries | *
> > http://mifos.org
> > >>>> <http://facebook.com/mifos> <http://www.twitter.com/mifos>
> > >>>>
> > >>
> >
> >
>
> --
> Karthik Krishnan
>