You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@fineract.apache.org by "Apoorva M.K" <ap...@gmail.com> on 2019/11/07 07:22:47 UTC
Re: [Mifos-developer] Scalability & Performance Working Group
Hi Ed,
I'm interested in helping out in this, I have gone through the list of next
steps. I will start working on what I can.
Regards,
Apoorva
On Thu 24 Oct, 2019, 1:57 AM Ed Cable, <ed...@mifos.org> wrote:
> Hi Mifos and Fineract communities,
>
> In follow up to my previous comments on Joseph's thread I wanted to
> initiate a collaborative community-wide effort to help address the ongoing
> and growing need for helping Mifos/Fineract scale and sustain high load
> environments.
>
> This really represent an area where we can demonstrate that together as
> Open Source Community we are much stronger than each individual or partner
> trying to tackle this alone.
>
> This would be for both Fineract and Fineract CN.
>
> *Why the need for this Working Group?*
> The formation of a working group focused on this subject is needed as
> evident from various mail thread or support requests trying to optimize the
> system for high load/volume environments and a growing number of prospects
> eager to use the platform but needing more visibility into its performance
> and ability to meet high TPS requirements.
>
> Secondly, there are many partner-led implementations and deployments of
> Fineract/Mifos supporting millions of clients and it would be valuable to
> share that knowledge across the community.
>
> Lastly, it's been quite some time since we've had public performance
> testing done, the efforts led by eSolve in 2017 (see wiki page) and IBM/Conflux
> in 2015
> <https://www.ibm.com/partnerworld/page/stg_ast_sys-mifos-x-on-ibm-powerlinux-servers>both
> pre-date the recent change from Hibernate to OpenJPA. Kumaranath with
> support from Avik of Fynarfin worked on performance related issues during
> 2018 GSOC -
> https://docs.google.com/document/d/18_awblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW_-In0
>
>
> *What/How*
>
> 1. Create a set of reproducible tools to enable automation of this on
> an ongoing basis.
> 1. Performance testing related data sets - i.e. DB dumps, scripts
> to generate test data
> 2. Postman scripts to run load tests
> 3. Perhaps we can create a repo on github for this?
> 2. Access to Resources
> 1. If you have hardware or cloud environments available to conduct
> these load/performance tests please share.
> 3. Culture & Transparency - Establish this as a priority area of the
> community to collectively address.
> 4. Documentation (likely on the Fineract Wiki) - I created this page
> to start: https://cwiki.apache.org/confluence/x/khD3Bw
> 1. Share and document existing results of performance testing to
> date. Share the details of environments you have set up to address high
> load needs.
> 2. Document the scenarios and TPS requirements that need to be
> tested
> 3. Document typical improvements to address performance (at a
> configuration, database, code, level, etc.)
> 5. Identify issues to fix
> 1. Log new issues, update existing issues, and tag appropriately.
> 6. Share code and fixes to address performance issues
> 1. Make these a priority and plan out in our release roadmapping.
>
> *Next Steps*
>
> - If interested in joining the group, respond to this thread.
> - Begin sharing your inputs on the wiki page.
> - Start fixing existing issues.
> - Help in creating and executing on our performance testing plan.
>
> Thanks,
>
> Ed
>
> Mifos-developer mailing list
> mifos-developer@lists.sourceforge.net
> Unsubscribe or change settings at:
> https://lists.sourceforge.net/lists/listinfo/mifos-developer
Re: [Mifos-developer] Scalability & Performance Working Group
Posted by Awasum Yannick <aw...@apache.org>.
+1
On Fri, Nov 8, 2019 at 9:02 AM Niklas Uhrberg <Ni...@resurs.se>
wrote:
> Nazeer, I think you sum important things up very well.
>
> When using the term "java layer" that can be scaled horizontally I simply
> meant the java microservices in Fineract CN or monolith java application in
> Fineract 1 excluding the database. Just to convey what I was actually
> thinking of, sorry if it was a bit vague.
>
> (The quote is "As previous posts have pointed out: since (or if ?) the
> java layer can be scaled horizontally it will probably not be a bottleneck.
> But since there is a common database just scaling the java layer may not be
> sufficient." )
>
> It boild down to that it is pointless to scale the application logic alone
> if the db becomes a bottleneck.
>
> I also think that having each microservice "own" its database in the sense
> that it is the only user of it makes a lot of sense. This is indeed that
> default when doing microservices. I suspect also that splitting the
> database is far more difficult than just splitting a monolith into
> multiple services.
>
> As I pointed out in the previous thread about NewSQL, there can be
> eventually consistent read models that contain data from multiple services
> serving the use cases where you effectively need data from multiple
> services in the same database. (For performance reasons.)
>
> Best regards Niklas
>
>
>
>
> Niklas Uhrberg
> Konsult
> IT Development | Loans & Credit
>
>
>
> Resurs Bank AB
>
>
>
> Växel: +46 42 38 20 00
> E-post: Niklas.Uhrberg@resurs.se
> Webb: www.resursbank.se
>
>
>
> ------------------------------
> *From:* Nazeer Hussain Shaik [nazeerhussain.shaik@gmail.com]
> *Sent:* Friday, November 08, 2019 7:03 AM
> *To:* dev@fineract.apache.org
> *Subject:* Re: [Mifos-developer] Scalability & Performance Working Group
>
> Hi Niklas,
>
> I agree with you we have to approach systematic way. Small correction.
> Java doesn't scale horizontally. Micro services should be scaled. Having a
> single instance of DB (single cluster with minimal read replicas) for all
> micro services will be a big bottleneck on production environments. Should
> be clustered for each micro services based on the load. Usually different
> micro services might have different DB types. For example one micro service
> might have Graph DB, another might have Mongo, etc.. As you know no
> architecture is perfect, we need to a take the approach which will give
> optimal benefits with some small drawbacks.
>
> Note: I am not saying for every deployment ,we need to have DB clusters
> for each micro service and load balancers. That depends upon the how much
> through put that deployment should take. But for client facing API(s) it
> should always better to have data aggregated and ready to serve the request
> immediately than constructing the data on the fly by reading different
> schema. If you need to have good performance it should always better to use
> eventual consistency in clustered environments. But some use cases might
> require strong consistency. So the configurations changes based on micro
> service to micro service.
>
> As all suggested, better to profile current implementation to find out
> where is the bottleneck.
>
> Regards,
> Nazeer
>
> On Fri, Nov 8, 2019 at 5:58 AM Tonio O <tc...@gmail.com> wrote:
>
>> I agree that performance tuning should be driven by non functional
>> requirements/goals and determined by performance testing of the
>> application. There is nothing wrong with the current technology stack if
>> developers adopt best practices and write efficient algorithms and
>> functions. Maybe a good place to start is to profile the application and
>> see where the bottlenecks exist(Memory, CPU, I/O). Identify which layer of
>> the applications needs to be scaled and how to scale both horizontally and
>> vertically.
>>
>>
>> On Thu, Nov 7, 2019 at 5:51 AM Niklas Uhrberg <Ni...@resurs.se>
>> wrote:
>>
>>> Hi all, I think the scalability question calls for a systematic and
>>> analytical way of working. First things to understand is what are the
>>> different functional parts that have individual scalability requirements,
>>> and then what is the behaviour/properties of the current implementation.
>>>
>>> Does the current implementation meet future requirements?
>>> If no (and it seems this is the case for at least some parts) understand
>>> what the limitations of the current architecture is. If the current
>>> architecture will not be able to support the requirements then it must
>>> change.
>>>
>>> As previous posts have pointed out: since (or if ?) the java layer can
>>> be scaled horizontally it will probably not be a bottleneck. But since
>>> there is a common database just scaling the java layer may not be
>>> sufficient.
>>>
>>> In any case setting up a potent test environment is the only way to
>>> really get the correct answers to questions about where the limits are.
>>>
>>> In previous thread there was a discussion about to what extent a NewSQL
>>> db could be used to increase the scalability. The only thing I want to say
>>> here is that there are use cases where by definition distributed
>>> transactions guaranteeing strong consistency and availability will never
>>> scale as well as eventual consistency architecture á la e.g. actor model
>>> with only aggregate level strong consistency guarantees.
>>>
>>> I think it is crucial to first identify the scalability requirements and
>>> understand how the system can be partitioned (including database) to meet
>>> the requirements and take the CAP theorem into account doing this.
>>>
>>> Best regards Niklas
>>>
>>>
>>>
>>>
>>>
>>> Niklas Uhrberg
>>> Konsult
>>> IT Development | Loans & Credit
>>>
>>>
>>>
>>> Resurs Bank AB
>>>
>>>
>>>
>>> Växel: +46 42 38 20 00
>>> E-post: Niklas.Uhrberg@resurs.se
>>> Webb: www.resursbank.se
>>>
>>>
>>>
>>> ------------------------------
>>> *From:* Giorgio Zoppi [giorgio.zoppi@gmail.com]
>>> *Sent:* Thursday, November 07, 2019 9:51 AM
>>> *To:* dev@fineract.apache.org
>>> *Cc:* Mifos software development
>>> *Subject:* Re: [Mifos-developer] Scalability & Performance Working Group
>>>
>>> Hi Nazeer,
>>> i was thinking about introduce Redis in the current architecture. Do you
>>> think it feasible?
>>> BR.
>>> Giorgio
>>>
>>> El jue., 7 nov. 2019 a las 9:24, Nazeer Hussain Shaik (<
>>> nazeerhussain.shaik@gmail.com>) escribió:
>>>
>>>> Hi all,
>>>>
>>>> Before thinking about moving micro services in different technologies
>>>> please think about getting the required skill set people and the
>>>> complexities around those technologies.
>>>> I believe scaling the micro services horizontally with proper hardware
>>>> configurations and by using some kind of load balance service (NGINX) may
>>>> be good idea.
>>>> In my current company we have more than 80 micro services and most of
>>>> them are written in Java technologies. We deploy our services on AWS
>>>> (Simulating SAAS model).
>>>> We use kubernetes for docker orchestration. Currently we are able to
>>>> scale 2 million requests. We are working on to scale to 4 million requests
>>>> at least. So I don't think Java is bottleneck here.
>>>> Please check whether the DB layer and JPA is performing as expected. I
>>>> believe you need to look into client facing API(s) instead of hitting the
>>>> direct micro services, better to use some kind of Indexers (Elastic Search,
>>>> etc..). Or aggregate the data in client facing services properly.
>>>>
>>>> Please correct me if my thinking is wrong.
>>>>
>>>> Regards,
>>>> Nazeer
>>>>
>>>> On Thu, Nov 7, 2019 at 12:54 PM Apoorva M.K <ap...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Ed,
>>>>> I'm interested in helping out in this, I have gone through the list of
>>>>> next steps. I will start working on what I can.
>>>>>
>>>>> Regards,
>>>>> Apoorva
>>>>>
>>>>> On Thu 24 Oct, 2019, 1:57 AM Ed Cable, <ed...@mifos.org> wrote:
>>>>>
>>>>>> Hi Mifos and Fineract communities,
>>>>>>
>>>>>> In follow up to my previous comments on Joseph's thread I wanted to
>>>>>> initiate a collaborative community-wide effort to help address the ongoing
>>>>>> and growing need for helping Mifos/Fineract scale and sustain high load
>>>>>> environments.
>>>>>>
>>>>>> This really represent an area where we can demonstrate that together
>>>>>> as Open Source Community we are much stronger than each individual or
>>>>>> partner trying to tackle this alone.
>>>>>>
>>>>>> This would be for both Fineract and Fineract CN.
>>>>>>
>>>>>> *Why the need for this Working Group?*
>>>>>> The formation of a working group focused on this subject is needed as
>>>>>> evident from various mail thread or support requests trying to optimize the
>>>>>> system for high load/volume environments and a growing number of prospects
>>>>>> eager to use the platform but needing more visibility into its performance
>>>>>> and ability to meet high TPS requirements.
>>>>>>
>>>>>> Secondly, there are many partner-led implementations and deployments
>>>>>> of Fineract/Mifos supporting millions of clients and it would be valuable
>>>>>> to share that knowledge across the community.
>>>>>>
>>>>>> Lastly, it's been quite some time since we've had public performance
>>>>>> testing done, the efforts led by eSolve in 2017 (see wiki page) and IBM/Conflux
>>>>>> in 2015
>>>>>> <https://www.ibm.com/partnerworld/page/stg_ast_sys-mifos-x-on-ibm-powerlinux-servers>both
>>>>>> pre-date the recent change from Hibernate to OpenJPA. Kumaranath with
>>>>>> support from Avik of Fynarfin worked on performance related issues during
>>>>>> 2018 GSOC -
>>>>>> https://docs.google.com/document/d/18_awblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW_-In0
>>>>>> <https://smex12-5-en-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fdocs.google.com%2fdocument%2fd%2f18%5fawblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW%5f%2dIn0&umid=ba389591-3c1d-4752-9f0f-b1566097d2f0&auth=5570038ba5c1d86c75907f538183887803b1409d-66acc5307e319e338515ff148c4394046f4e061b>
>>>>>>
>>>>>>
>>>>>> *What/How*
>>>>>>
>>>>>> 1. Create a set of reproducible tools to enable automation of
>>>>>> this on an ongoing basis.
>>>>>> 1. Performance testing related data sets - i.e. DB dumps,
>>>>>> scripts to generate test data
>>>>>> 2. Postman scripts to run load tests
>>>>>> 3. Perhaps we can create a repo on github for this?
>>>>>> 2. Access to Resources
>>>>>> 1. If you have hardware or cloud environments available to
>>>>>> conduct these load/performance tests please share.
>>>>>> 3. Culture & Transparency - Establish this as a priority area of
>>>>>> the community to collectively address.
>>>>>> 4. Documentation (likely on the Fineract Wiki) - I created this
>>>>>> page to start: https://cwiki.apache.org/confluence/x/khD3Bw
>>>>>> 1. Share and document existing results of performance testing
>>>>>> to date. Share the details of environments you have set up to address high
>>>>>> load needs.
>>>>>> 2. Document the scenarios and TPS requirements that need to be
>>>>>> tested
>>>>>> 3. Document typical improvements to address performance (at a
>>>>>> configuration, database, code, level, etc.)
>>>>>> 5. Identify issues to fix
>>>>>> 1. Log new issues, update existing issues, and tag
>>>>>> appropriately.
>>>>>> 6. Share code and fixes to address performance issues
>>>>>> 1. Make these a priority and plan out in our release
>>>>>> roadmapping.
>>>>>>
>>>>>> *Next Steps*
>>>>>>
>>>>>> - If interested in joining the group, respond to this thread.
>>>>>> - Begin sharing your inputs on the wiki page.
>>>>>> - Start fixing existing issues.
>>>>>> - Help in creating and executing on our performance testing plan.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Ed
>>>>>>
>>>>>> Mifos-developer mailing list
>>>>>> mifos-developer@lists.sourceforge.net
>>>>>> Unsubscribe or change settings at:
>>>>>> https://lists.sourceforge.net/lists/listinfo/mifos-developer
>>>>>
>>>>>
>>>
>>> --
>>> Life is a chess game - Anonymous.
>>>
>>>
RE: [Mifos-developer] Scalability & Performance Working Group
Posted by Niklas Uhrberg <Ni...@resurs.se>.
Nazeer, I think you sum important things up very well.
When using the term "java layer" that can be scaled horizontally I simply meant the java microservices in Fineract CN or monolith java application in Fineract 1 excluding the database. Just to convey what I was actually thinking of, sorry if it was a bit vague.
(The quote is "As previous posts have pointed out: since (or if ?) the java layer can be scaled horizontally it will probably not be a bottleneck. But since there is a common database just scaling the java layer may not be sufficient." )
It boild down to that it is pointless to scale the application logic alone if the db becomes a bottleneck.
I also think that having each microservice "own" its database in the sense that it is the only user of it makes a lot of sense. This is indeed that default when doing microservices. I suspect also that splitting the database is far more difficult than just splitting a monolith into multiple services.
As I pointed out in the previous thread about NewSQL, there can be eventually consistent read models that contain data from multiple services serving the use cases where you effectively need data from multiple services in the same database. (For performance reasons.)
Best regards Niklas
[cid:image88abc9.PNG@bfdc9cf2.4a9c1997]
Niklas Uhrberg
Konsult
IT Development | Loans & Credit
Resurs Bank AB
Växel: +46 42 38 20 00
E-post: Niklas.Uhrberg@resurs.se<ma...@resurs.se>
Webb: www.resursbank.se<http://www.resursbank.se>
________________________________
From: Nazeer Hussain Shaik [nazeerhussain.shaik@gmail.com]
Sent: Friday, November 08, 2019 7:03 AM
To: dev@fineract.apache.org
Subject: Re: [Mifos-developer] Scalability & Performance Working Group
Hi Niklas,
I agree with you we have to approach systematic way. Small correction. Java doesn't scale horizontally. Micro services should be scaled. Having a single instance of DB (single cluster with minimal read replicas) for all micro services will be a big bottleneck on production environments. Should be clustered for each micro services based on the load. Usually different micro services might have different DB types. For example one micro service might have Graph DB, another might have Mongo, etc.. As you know no architecture is perfect, we need to a take the approach which will give optimal benefits with some small drawbacks.
Note: I am not saying for every deployment ,we need to have DB clusters for each micro service and load balancers. That depends upon the how much through put that deployment should take. But for client facing API(s) it should always better to have data aggregated and ready to serve the request immediately than constructing the data on the fly by reading different schema. If you need to have good performance it should always better to use eventual consistency in clustered environments. But some use cases might require strong consistency. So the configurations changes based on micro service to micro service.
As all suggested, better to profile current implementation to find out where is the bottleneck.
Regards,
Nazeer
On Fri, Nov 8, 2019 at 5:58 AM Tonio O <tc...@gmail.com>> wrote:
I agree that performance tuning should be driven by non functional requirements/goals and determined by performance testing of the application. There is nothing wrong with the current technology stack if developers adopt best practices and write efficient algorithms and functions. Maybe a good place to start is to profile the application and see where the bottlenecks exist(Memory, CPU, I/O). Identify which layer of the applications needs to be scaled and how to scale both horizontally and vertically.
On Thu, Nov 7, 2019 at 5:51 AM Niklas Uhrberg <Ni...@resurs.se>> wrote:
Hi all, I think the scalability question calls for a systematic and analytical way of working. First things to understand is what are the different functional parts that have individual scalability requirements, and then what is the behaviour/properties of the current implementation.
Does the current implementation meet future requirements?
If no (and it seems this is the case for at least some parts) understand what the limitations of the current architecture is. If the current architecture will not be able to support the requirements then it must change.
As previous posts have pointed out: since (or if ?) the java layer can be scaled horizontally it will probably not be a bottleneck. But since there is a common database just scaling the java layer may not be sufficient.
In any case setting up a potent test environment is the only way to really get the correct answers to questions about where the limits are.
In previous thread there was a discussion about to what extent a NewSQL db could be used to increase the scalability. The only thing I want to say here is that there are use cases where by definition distributed transactions guaranteeing strong consistency and availability will never scale as well as eventual consistency architecture á la e.g. actor model with only aggregate level strong consistency guarantees.
I think it is crucial to first identify the scalability requirements and understand how the system can be partitioned (including database) to meet the requirements and take the CAP theorem into account doing this.
Best regards Niklas
[cid:16e485f956838ca05f11]
Niklas Uhrberg
Konsult
IT Development | Loans & Credit
Resurs Bank AB
Växel: +46 42 38 20 00
E-post: Niklas.Uhrberg@resurs.se<ma...@resurs.se>
Webb: www.resursbank.se<http://www.resursbank.se>
________________________________
From: Giorgio Zoppi [giorgio.zoppi@gmail.com<ma...@gmail.com>]
Sent: Thursday, November 07, 2019 9:51 AM
To: dev@fineract.apache.org<ma...@fineract.apache.org>
Cc: Mifos software development
Subject: Re: [Mifos-developer] Scalability & Performance Working Group
Hi Nazeer,
i was thinking about introduce Redis in the current architecture. Do you think it feasible?
BR.
Giorgio
El jue., 7 nov. 2019 a las 9:24, Nazeer Hussain Shaik (<na...@gmail.com>>) escribió:
Hi all,
Before thinking about moving micro services in different technologies please think about getting the required skill set people and the complexities around those technologies.
I believe scaling the micro services horizontally with proper hardware configurations and by using some kind of load balance service (NGINX) may be good idea.
In my current company we have more than 80 micro services and most of them are written in Java technologies. We deploy our services on AWS (Simulating SAAS model).
We use kubernetes for docker orchestration. Currently we are able to scale 2 million requests. We are working on to scale to 4 million requests at least. So I don't think Java is bottleneck here.
Please check whether the DB layer and JPA is performing as expected. I believe you need to look into client facing API(s) instead of hitting the direct micro services, better to use some kind of Indexers (Elastic Search, etc..). Or aggregate the data in client facing services properly.
Please correct me if my thinking is wrong.
Regards,
Nazeer
On Thu, Nov 7, 2019 at 12:54 PM Apoorva M.K <ap...@gmail.com>> wrote:
Hi Ed,
I'm interested in helping out in this, I have gone through the list of next steps. I will start working on what I can.
Regards,
Apoorva
On Thu 24 Oct, 2019, 1:57 AM Ed Cable, <ed...@mifos.org>> wrote:
Hi Mifos and Fineract communities,
In follow up to my previous comments on Joseph's thread I wanted to initiate a collaborative community-wide effort to help address the ongoing and growing need for helping Mifos/Fineract scale and sustain high load environments.
This really represent an area where we can demonstrate that together as Open Source Community we are much stronger than each individual or partner trying to tackle this alone.
This would be for both Fineract and Fineract CN.
Why the need for this Working Group?
The formation of a working group focused on this subject is needed as evident from various mail thread or support requests trying to optimize the system for high load/volume environments and a growing number of prospects eager to use the platform but needing more visibility into its performance and ability to meet high TPS requirements.
Secondly, there are many partner-led implementations and deployments of Fineract/Mifos supporting millions of clients and it would be valuable to share that knowledge across the community.
Lastly, it's been quite some time since we've had public performance testing done, the efforts led by eSolve in 2017 (see wiki page) and IBM/Conflux in 2015 <https://www.ibm.com/partnerworld/page/stg_ast_sys-mifos-x-on-ibm-powerlinux-servers> both pre-date the recent change from Hibernate to OpenJPA. Kumaranath with support from Avik of Fynarfin worked on performance related issues during 2018 GSOC - https://docs.google.com/document/d/18_awblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW_-In0<https://smex12-5-en-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fdocs.google.com%2fdocument%2fd%2f18%5fawblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW%5f%2dIn0&umid=ba389591-3c1d-4752-9f0f-b1566097d2f0&auth=5570038ba5c1d86c75907f538183887803b1409d-66acc5307e319e338515ff148c4394046f4e061b>
What/How
1. Create a set of reproducible tools to enable automation of this on an ongoing basis.
* Performance testing related data sets - i.e. DB dumps, scripts to generate test data
* Postman scripts to run load tests
* Perhaps we can create a repo on github for this?
2. Access to Resources
* If you have hardware or cloud environments available to conduct these load/performance tests please share.
3. Culture & Transparency - Establish this as a priority area of the community to collectively address.
4. Documentation (likely on the Fineract Wiki) - I created this page to start: https://cwiki.apache.org/confluence/x/khD3Bw
* Share and document existing results of performance testing to date. Share the details of environments you have set up to address high load needs.
* Document the scenarios and TPS requirements that need to be tested
* Document typical improvements to address performance (at a configuration, database, code, level, etc.)
5. Identify issues to fix
* Log new issues, update existing issues, and tag appropriately.
6. Share code and fixes to address performance issues
* Make these a priority and plan out in our release roadmapping.
Next Steps
* If interested in joining the group, respond to this thread.
* Begin sharing your inputs on the wiki page.
* Start fixing existing issues.
* Help in creating and executing on our performance testing plan.
Thanks,
Ed
Mifos-developer mailing list
mifos-developer@lists.sourceforge.net<ma...@lists.sourceforge.net>
Unsubscribe or change settings at:
https://lists.sourceforge.net/lists/listinfo/mifos-developer
--
Life is a chess game - Anonymous.
Re: [Mifos-developer] Scalability & Performance Working Group
Posted by Nazeer Hussain Shaik <na...@gmail.com>.
Hi Niklas,
I agree with you we have to approach systematic way. Small correction. Java
doesn't scale horizontally. Micro services should be scaled. Having a
single instance of DB (single cluster with minimal read replicas) for all
micro services will be a big bottleneck on production environments. Should
be clustered for each micro services based on the load. Usually different
micro services might have different DB types. For example one micro service
might have Graph DB, another might have Mongo, etc.. As you know no
architecture is perfect, we need to a take the approach which will give
optimal benefits with some small drawbacks.
Note: I am not saying for every deployment ,we need to have DB clusters for
each micro service and load balancers. That depends upon the how much
through put that deployment should take. But for client facing API(s) it
should always better to have data aggregated and ready to serve the request
immediately than constructing the data on the fly by reading different
schema. If you need to have good performance it should always better to use
eventual consistency in clustered environments. But some use cases might
require strong consistency. So the configurations changes based on micro
service to micro service.
As all suggested, better to profile current implementation to find out
where is the bottleneck.
Regards,
Nazeer
On Fri, Nov 8, 2019 at 5:58 AM Tonio O <tc...@gmail.com> wrote:
> I agree that performance tuning should be driven by non functional
> requirements/goals and determined by performance testing of the
> application. There is nothing wrong with the current technology stack if
> developers adopt best practices and write efficient algorithms and
> functions. Maybe a good place to start is to profile the application and
> see where the bottlenecks exist(Memory, CPU, I/O). Identify which layer of
> the applications needs to be scaled and how to scale both horizontally and
> vertically.
>
>
> On Thu, Nov 7, 2019 at 5:51 AM Niklas Uhrberg <Ni...@resurs.se>
> wrote:
>
>> Hi all, I think the scalability question calls for a systematic and
>> analytical way of working. First things to understand is what are the
>> different functional parts that have individual scalability requirements,
>> and then what is the behaviour/properties of the current implementation.
>>
>> Does the current implementation meet future requirements?
>> If no (and it seems this is the case for at least some parts) understand
>> what the limitations of the current architecture is. If the current
>> architecture will not be able to support the requirements then it must
>> change.
>>
>> As previous posts have pointed out: since (or if ?) the java layer can be
>> scaled horizontally it will probably not be a bottleneck. But since there
>> is a common database just scaling the java layer may not be sufficient.
>>
>> In any case setting up a potent test environment is the only way to
>> really get the correct answers to questions about where the limits are.
>>
>> In previous thread there was a discussion about to what extent a NewSQL
>> db could be used to increase the scalability. The only thing I want to say
>> here is that there are use cases where by definition distributed
>> transactions guaranteeing strong consistency and availability will never
>> scale as well as eventual consistency architecture á la e.g. actor model
>> with only aggregate level strong consistency guarantees.
>>
>> I think it is crucial to first identify the scalability requirements and
>> understand how the system can be partitioned (including database) to meet
>> the requirements and take the CAP theorem into account doing this.
>>
>> Best regards Niklas
>>
>>
>>
>>
>>
>> Niklas Uhrberg
>> Konsult
>> IT Development | Loans & Credit
>>
>>
>>
>> Resurs Bank AB
>>
>>
>>
>> Växel: +46 42 38 20 00
>> E-post: Niklas.Uhrberg@resurs.se
>> Webb: www.resursbank.se
>>
>>
>>
>> ------------------------------
>> *From:* Giorgio Zoppi [giorgio.zoppi@gmail.com]
>> *Sent:* Thursday, November 07, 2019 9:51 AM
>> *To:* dev@fineract.apache.org
>> *Cc:* Mifos software development
>> *Subject:* Re: [Mifos-developer] Scalability & Performance Working Group
>>
>> Hi Nazeer,
>> i was thinking about introduce Redis in the current architecture. Do you
>> think it feasible?
>> BR.
>> Giorgio
>>
>> El jue., 7 nov. 2019 a las 9:24, Nazeer Hussain Shaik (<
>> nazeerhussain.shaik@gmail.com>) escribió:
>>
>>> Hi all,
>>>
>>> Before thinking about moving micro services in different technologies
>>> please think about getting the required skill set people and the
>>> complexities around those technologies.
>>> I believe scaling the micro services horizontally with proper hardware
>>> configurations and by using some kind of load balance service (NGINX) may
>>> be good idea.
>>> In my current company we have more than 80 micro services and most of
>>> them are written in Java technologies. We deploy our services on AWS
>>> (Simulating SAAS model).
>>> We use kubernetes for docker orchestration. Currently we are able to
>>> scale 2 million requests. We are working on to scale to 4 million requests
>>> at least. So I don't think Java is bottleneck here.
>>> Please check whether the DB layer and JPA is performing as expected. I
>>> believe you need to look into client facing API(s) instead of hitting the
>>> direct micro services, better to use some kind of Indexers (Elastic Search,
>>> etc..). Or aggregate the data in client facing services properly.
>>>
>>> Please correct me if my thinking is wrong.
>>>
>>> Regards,
>>> Nazeer
>>>
>>> On Thu, Nov 7, 2019 at 12:54 PM Apoorva M.K <ap...@gmail.com>
>>> wrote:
>>>
>>>> Hi Ed,
>>>> I'm interested in helping out in this, I have gone through the list of
>>>> next steps. I will start working on what I can.
>>>>
>>>> Regards,
>>>> Apoorva
>>>>
>>>> On Thu 24 Oct, 2019, 1:57 AM Ed Cable, <ed...@mifos.org> wrote:
>>>>
>>>>> Hi Mifos and Fineract communities,
>>>>>
>>>>> In follow up to my previous comments on Joseph's thread I wanted to
>>>>> initiate a collaborative community-wide effort to help address the ongoing
>>>>> and growing need for helping Mifos/Fineract scale and sustain high load
>>>>> environments.
>>>>>
>>>>> This really represent an area where we can demonstrate that together
>>>>> as Open Source Community we are much stronger than each individual or
>>>>> partner trying to tackle this alone.
>>>>>
>>>>> This would be for both Fineract and Fineract CN.
>>>>>
>>>>> *Why the need for this Working Group?*
>>>>> The formation of a working group focused on this subject is needed as
>>>>> evident from various mail thread or support requests trying to optimize the
>>>>> system for high load/volume environments and a growing number of prospects
>>>>> eager to use the platform but needing more visibility into its performance
>>>>> and ability to meet high TPS requirements.
>>>>>
>>>>> Secondly, there are many partner-led implementations and deployments
>>>>> of Fineract/Mifos supporting millions of clients and it would be valuable
>>>>> to share that knowledge across the community.
>>>>>
>>>>> Lastly, it's been quite some time since we've had public performance
>>>>> testing done, the efforts led by eSolve in 2017 (see wiki page) and IBM/Conflux
>>>>> in 2015
>>>>> <https://www.ibm.com/partnerworld/page/stg_ast_sys-mifos-x-on-ibm-powerlinux-servers>both
>>>>> pre-date the recent change from Hibernate to OpenJPA. Kumaranath with
>>>>> support from Avik of Fynarfin worked on performance related issues during
>>>>> 2018 GSOC -
>>>>> https://docs.google.com/document/d/18_awblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW_-In0
>>>>> <https://smex12-5-en-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fdocs.google.com%2fdocument%2fd%2f18%5fawblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW%5f%2dIn0&umid=ba389591-3c1d-4752-9f0f-b1566097d2f0&auth=5570038ba5c1d86c75907f538183887803b1409d-66acc5307e319e338515ff148c4394046f4e061b>
>>>>>
>>>>>
>>>>> *What/How*
>>>>>
>>>>> 1. Create a set of reproducible tools to enable automation of this
>>>>> on an ongoing basis.
>>>>> 1. Performance testing related data sets - i.e. DB dumps,
>>>>> scripts to generate test data
>>>>> 2. Postman scripts to run load tests
>>>>> 3. Perhaps we can create a repo on github for this?
>>>>> 2. Access to Resources
>>>>> 1. If you have hardware or cloud environments available to
>>>>> conduct these load/performance tests please share.
>>>>> 3. Culture & Transparency - Establish this as a priority area of
>>>>> the community to collectively address.
>>>>> 4. Documentation (likely on the Fineract Wiki) - I created this
>>>>> page to start: https://cwiki.apache.org/confluence/x/khD3Bw
>>>>> 1. Share and document existing results of performance testing
>>>>> to date. Share the details of environments you have set up to address high
>>>>> load needs.
>>>>> 2. Document the scenarios and TPS requirements that need to be
>>>>> tested
>>>>> 3. Document typical improvements to address performance (at a
>>>>> configuration, database, code, level, etc.)
>>>>> 5. Identify issues to fix
>>>>> 1. Log new issues, update existing issues, and tag
>>>>> appropriately.
>>>>> 6. Share code and fixes to address performance issues
>>>>> 1. Make these a priority and plan out in our release
>>>>> roadmapping.
>>>>>
>>>>> *Next Steps*
>>>>>
>>>>> - If interested in joining the group, respond to this thread.
>>>>> - Begin sharing your inputs on the wiki page.
>>>>> - Start fixing existing issues.
>>>>> - Help in creating and executing on our performance testing plan.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Ed
>>>>>
>>>>> Mifos-developer mailing list
>>>>> mifos-developer@lists.sourceforge.net
>>>>> Unsubscribe or change settings at:
>>>>> https://lists.sourceforge.net/lists/listinfo/mifos-developer
>>>>
>>>>
>>
>> --
>> Life is a chess game - Anonymous.
>>
>>
Re: [Mifos-developer] Scalability & Performance Working Group
Posted by Tonio O <tc...@gmail.com>.
I agree that performance tuning should be driven by non functional
requirements/goals and determined by performance testing of the
application. There is nothing wrong with the current technology stack if
developers adopt best practices and write efficient algorithms and
functions. Maybe a good place to start is to profile the application and
see where the bottlenecks exist(Memory, CPU, I/O). Identify which layer of
the applications needs to be scaled and how to scale both horizontally and
vertically.
On Thu, Nov 7, 2019 at 5:51 AM Niklas Uhrberg <Ni...@resurs.se>
wrote:
> Hi all, I think the scalability question calls for a systematic and
> analytical way of working. First things to understand is what are the
> different functional parts that have individual scalability requirements,
> and then what is the behaviour/properties of the current implementation.
>
> Does the current implementation meet future requirements?
> If no (and it seems this is the case for at least some parts) understand
> what the limitations of the current architecture is. If the current
> architecture will not be able to support the requirements then it must
> change.
>
> As previous posts have pointed out: since (or if ?) the java layer can be
> scaled horizontally it will probably not be a bottleneck. But since there
> is a common database just scaling the java layer may not be sufficient.
>
> In any case setting up a potent test environment is the only way to really
> get the correct answers to questions about where the limits are.
>
> In previous thread there was a discussion about to what extent a NewSQL db
> could be used to increase the scalability. The only thing I want to say
> here is that there are use cases where by definition distributed
> transactions guaranteeing strong consistency and availability will never
> scale as well as eventual consistency architecture á la e.g. actor model
> with only aggregate level strong consistency guarantees.
>
> I think it is crucial to first identify the scalability requirements and
> understand how the system can be partitioned (including database) to meet
> the requirements and take the CAP theorem into account doing this.
>
> Best regards Niklas
>
>
>
>
>
> Niklas Uhrberg
> Konsult
> IT Development | Loans & Credit
>
>
>
> Resurs Bank AB
>
>
>
> Växel: +46 42 38 20 00
> E-post: Niklas.Uhrberg@resurs.se
> Webb: www.resursbank.se
>
>
>
> ------------------------------
> *From:* Giorgio Zoppi [giorgio.zoppi@gmail.com]
> *Sent:* Thursday, November 07, 2019 9:51 AM
> *To:* dev@fineract.apache.org
> *Cc:* Mifos software development
> *Subject:* Re: [Mifos-developer] Scalability & Performance Working Group
>
> Hi Nazeer,
> i was thinking about introduce Redis in the current architecture. Do you
> think it feasible?
> BR.
> Giorgio
>
> El jue., 7 nov. 2019 a las 9:24, Nazeer Hussain Shaik (<
> nazeerhussain.shaik@gmail.com>) escribió:
>
>> Hi all,
>>
>> Before thinking about moving micro services in different technologies
>> please think about getting the required skill set people and the
>> complexities around those technologies.
>> I believe scaling the micro services horizontally with proper hardware
>> configurations and by using some kind of load balance service (NGINX) may
>> be good idea.
>> In my current company we have more than 80 micro services and most of
>> them are written in Java technologies. We deploy our services on AWS
>> (Simulating SAAS model).
>> We use kubernetes for docker orchestration. Currently we are able to
>> scale 2 million requests. We are working on to scale to 4 million requests
>> at least. So I don't think Java is bottleneck here.
>> Please check whether the DB layer and JPA is performing as expected. I
>> believe you need to look into client facing API(s) instead of hitting the
>> direct micro services, better to use some kind of Indexers (Elastic Search,
>> etc..). Or aggregate the data in client facing services properly.
>>
>> Please correct me if my thinking is wrong.
>>
>> Regards,
>> Nazeer
>>
>> On Thu, Nov 7, 2019 at 12:54 PM Apoorva M.K <ap...@gmail.com>
>> wrote:
>>
>>> Hi Ed,
>>> I'm interested in helping out in this, I have gone through the list of
>>> next steps. I will start working on what I can.
>>>
>>> Regards,
>>> Apoorva
>>>
>>> On Thu 24 Oct, 2019, 1:57 AM Ed Cable, <ed...@mifos.org> wrote:
>>>
>>>> Hi Mifos and Fineract communities,
>>>>
>>>> In follow up to my previous comments on Joseph's thread I wanted to
>>>> initiate a collaborative community-wide effort to help address the ongoing
>>>> and growing need for helping Mifos/Fineract scale and sustain high load
>>>> environments.
>>>>
>>>> This really represent an area where we can demonstrate that together as
>>>> Open Source Community we are much stronger than each individual or partner
>>>> trying to tackle this alone.
>>>>
>>>> This would be for both Fineract and Fineract CN.
>>>>
>>>> *Why the need for this Working Group?*
>>>> The formation of a working group focused on this subject is needed as
>>>> evident from various mail thread or support requests trying to optimize the
>>>> system for high load/volume environments and a growing number of prospects
>>>> eager to use the platform but needing more visibility into its performance
>>>> and ability to meet high TPS requirements.
>>>>
>>>> Secondly, there are many partner-led implementations and deployments of
>>>> Fineract/Mifos supporting millions of clients and it would be valuable to
>>>> share that knowledge across the community.
>>>>
>>>> Lastly, it's been quite some time since we've had public performance
>>>> testing done, the efforts led by eSolve in 2017 (see wiki page) and IBM/Conflux
>>>> in 2015
>>>> <https://www.ibm.com/partnerworld/page/stg_ast_sys-mifos-x-on-ibm-powerlinux-servers>both
>>>> pre-date the recent change from Hibernate to OpenJPA. Kumaranath with
>>>> support from Avik of Fynarfin worked on performance related issues during
>>>> 2018 GSOC -
>>>> https://docs.google.com/document/d/18_awblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW_-In0
>>>> <https://smex12-5-en-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fdocs.google.com%2fdocument%2fd%2f18%5fawblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW%5f%2dIn0&umid=ba389591-3c1d-4752-9f0f-b1566097d2f0&auth=5570038ba5c1d86c75907f538183887803b1409d-66acc5307e319e338515ff148c4394046f4e061b>
>>>>
>>>>
>>>> *What/How*
>>>>
>>>> 1. Create a set of reproducible tools to enable automation of this
>>>> on an ongoing basis.
>>>> 1. Performance testing related data sets - i.e. DB dumps,
>>>> scripts to generate test data
>>>> 2. Postman scripts to run load tests
>>>> 3. Perhaps we can create a repo on github for this?
>>>> 2. Access to Resources
>>>> 1. If you have hardware or cloud environments available to
>>>> conduct these load/performance tests please share.
>>>> 3. Culture & Transparency - Establish this as a priority area of
>>>> the community to collectively address.
>>>> 4. Documentation (likely on the Fineract Wiki) - I created this
>>>> page to start: https://cwiki.apache.org/confluence/x/khD3Bw
>>>> 1. Share and document existing results of performance testing to
>>>> date. Share the details of environments you have set up to address high
>>>> load needs.
>>>> 2. Document the scenarios and TPS requirements that need to be
>>>> tested
>>>> 3. Document typical improvements to address performance (at a
>>>> configuration, database, code, level, etc.)
>>>> 5. Identify issues to fix
>>>> 1. Log new issues, update existing issues, and tag
>>>> appropriately.
>>>> 6. Share code and fixes to address performance issues
>>>> 1. Make these a priority and plan out in our release
>>>> roadmapping.
>>>>
>>>> *Next Steps*
>>>>
>>>> - If interested in joining the group, respond to this thread.
>>>> - Begin sharing your inputs on the wiki page.
>>>> - Start fixing existing issues.
>>>> - Help in creating and executing on our performance testing plan.
>>>>
>>>> Thanks,
>>>>
>>>> Ed
>>>>
>>>> Mifos-developer mailing list
>>>> mifos-developer@lists.sourceforge.net
>>>> Unsubscribe or change settings at:
>>>> https://lists.sourceforge.net/lists/listinfo/mifos-developer
>>>
>>>
>
> --
> Life is a chess game - Anonymous.
>
>
RE: [Mifos-developer] Scalability & Performance Working Group
Posted by Niklas Uhrberg <Ni...@resurs.se>.
Hi all, I think the scalability question calls for a systematic and analytical way of working. First things to understand is what are the different functional parts that have individual scalability requirements, and then what is the behaviour/properties of the current implementation.
Does the current implementation meet future requirements?
If no (and it seems this is the case for at least some parts) understand what the limitations of the current architecture is. If the current architecture will not be able to support the requirements then it must change.
As previous posts have pointed out: since (or if ?) the java layer can be scaled horizontally it will probably not be a bottleneck. But since there is a common database just scaling the java layer may not be sufficient.
In any case setting up a potent test environment is the only way to really get the correct answers to questions about where the limits are.
In previous thread there was a discussion about to what extent a NewSQL db could be used to increase the scalability. The only thing I want to say here is that there are use cases where by definition distributed transactions guaranteeing strong consistency and availability will never scale as well as eventual consistency architecture á la e.g. actor model with only aggregate level strong consistency guarantees.
I think it is crucial to first identify the scalability requirements and understand how the system can be partitioned (including database) to meet the requirements and take the CAP theorem into account doing this.
Best regards Niklas
[cid:image3081af.PNG@0cd9fcd9.4a930b7a]
Niklas Uhrberg
Konsult
IT Development | Loans & Credit
Resurs Bank AB
Växel: +46 42 38 20 00
E-post: Niklas.Uhrberg@resurs.se<ma...@resurs.se>
Webb: www.resursbank.se<http://www.resursbank.se>
________________________________
From: Giorgio Zoppi [giorgio.zoppi@gmail.com]
Sent: Thursday, November 07, 2019 9:51 AM
To: dev@fineract.apache.org
Cc: Mifos software development
Subject: Re: [Mifos-developer] Scalability & Performance Working Group
Hi Nazeer,
i was thinking about introduce Redis in the current architecture. Do you think it feasible?
BR.
Giorgio
El jue., 7 nov. 2019 a las 9:24, Nazeer Hussain Shaik (<na...@gmail.com>>) escribió:
Hi all,
Before thinking about moving micro services in different technologies please think about getting the required skill set people and the complexities around those technologies.
I believe scaling the micro services horizontally with proper hardware configurations and by using some kind of load balance service (NGINX) may be good idea.
In my current company we have more than 80 micro services and most of them are written in Java technologies. We deploy our services on AWS (Simulating SAAS model).
We use kubernetes for docker orchestration. Currently we are able to scale 2 million requests. We are working on to scale to 4 million requests at least. So I don't think Java is bottleneck here.
Please check whether the DB layer and JPA is performing as expected. I believe you need to look into client facing API(s) instead of hitting the direct micro services, better to use some kind of Indexers (Elastic Search, etc..). Or aggregate the data in client facing services properly.
Please correct me if my thinking is wrong.
Regards,
Nazeer
On Thu, Nov 7, 2019 at 12:54 PM Apoorva M.K <ap...@gmail.com>> wrote:
Hi Ed,
I'm interested in helping out in this, I have gone through the list of next steps. I will start working on what I can.
Regards,
Apoorva
On Thu 24 Oct, 2019, 1:57 AM Ed Cable, <ed...@mifos.org>> wrote:
Hi Mifos and Fineract communities,
In follow up to my previous comments on Joseph's thread I wanted to initiate a collaborative community-wide effort to help address the ongoing and growing need for helping Mifos/Fineract scale and sustain high load environments.
This really represent an area where we can demonstrate that together as Open Source Community we are much stronger than each individual or partner trying to tackle this alone.
This would be for both Fineract and Fineract CN.
Why the need for this Working Group?
The formation of a working group focused on this subject is needed as evident from various mail thread or support requests trying to optimize the system for high load/volume environments and a growing number of prospects eager to use the platform but needing more visibility into its performance and ability to meet high TPS requirements.
Secondly, there are many partner-led implementations and deployments of Fineract/Mifos supporting millions of clients and it would be valuable to share that knowledge across the community.
Lastly, it's been quite some time since we've had public performance testing done, the efforts led by eSolve in 2017 (see wiki page) and IBM/Conflux in 2015 <https://www.ibm.com/partnerworld/page/stg_ast_sys-mifos-x-on-ibm-powerlinux-servers> both pre-date the recent change from Hibernate to OpenJPA. Kumaranath with support from Avik of Fynarfin worked on performance related issues during 2018 GSOC - https://docs.google.com/document/d/18_awblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW_-In0<https://smex12-5-en-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fdocs.google.com%2fdocument%2fd%2f18%5fawblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW%5f%2dIn0&umid=ba389591-3c1d-4752-9f0f-b1566097d2f0&auth=5570038ba5c1d86c75907f538183887803b1409d-66acc5307e319e338515ff148c4394046f4e061b>
What/How
1. Create a set of reproducible tools to enable automation of this on an ongoing basis.
* Performance testing related data sets - i.e. DB dumps, scripts to generate test data
* Postman scripts to run load tests
* Perhaps we can create a repo on github for this?
2. Access to Resources
* If you have hardware or cloud environments available to conduct these load/performance tests please share.
3. Culture & Transparency - Establish this as a priority area of the community to collectively address.
4. Documentation (likely on the Fineract Wiki) - I created this page to start: https://cwiki.apache.org/confluence/x/khD3Bw
* Share and document existing results of performance testing to date. Share the details of environments you have set up to address high load needs.
* Document the scenarios and TPS requirements that need to be tested
* Document typical improvements to address performance (at a configuration, database, code, level, etc.)
5. Identify issues to fix
* Log new issues, update existing issues, and tag appropriately.
6. Share code and fixes to address performance issues
* Make these a priority and plan out in our release roadmapping.
Next Steps
* If interested in joining the group, respond to this thread.
* Begin sharing your inputs on the wiki page.
* Start fixing existing issues.
* Help in creating and executing on our performance testing plan.
Thanks,
Ed
Mifos-developer mailing list
mifos-developer@lists.sourceforge.net<ma...@lists.sourceforge.net>
Unsubscribe or change settings at:
https://lists.sourceforge.net/lists/listinfo/mifos-developer
--
Life is a chess game - Anonymous.
Re: [Mifos-developer] Scalability & Performance Working Group
Posted by Giorgio Zoppi <gi...@gmail.com>.
Hi Nazeer,
i was thinking about introduce Redis in the current architecture. Do you
think it feasible?
BR.
Giorgio
El jue., 7 nov. 2019 a las 9:24, Nazeer Hussain Shaik (<
nazeerhussain.shaik@gmail.com>) escribió:
> Hi all,
>
> Before thinking about moving micro services in different technologies
> please think about getting the required skill set people and the
> complexities around those technologies.
> I believe scaling the micro services horizontally with proper hardware
> configurations and by using some kind of load balance service (NGINX) may
> be good idea.
> In my current company we have more than 80 micro services and most of them
> are written in Java technologies. We deploy our services on AWS (Simulating
> SAAS model).
> We use kubernetes for docker orchestration. Currently we are able to scale
> 2 million requests. We are working on to scale to 4 million requests at
> least. So I don't think Java is bottleneck here.
> Please check whether the DB layer and JPA is performing as expected. I
> believe you need to look into client facing API(s) instead of hitting the
> direct micro services, better to use some kind of Indexers (Elastic Search,
> etc..). Or aggregate the data in client facing services properly.
>
> Please correct me if my thinking is wrong.
>
> Regards,
> Nazeer
>
> On Thu, Nov 7, 2019 at 12:54 PM Apoorva M.K <ap...@gmail.com>
> wrote:
>
>> Hi Ed,
>> I'm interested in helping out in this, I have gone through the list of
>> next steps. I will start working on what I can.
>>
>> Regards,
>> Apoorva
>>
>> On Thu 24 Oct, 2019, 1:57 AM Ed Cable, <ed...@mifos.org> wrote:
>>
>>> Hi Mifos and Fineract communities,
>>>
>>> In follow up to my previous comments on Joseph's thread I wanted to
>>> initiate a collaborative community-wide effort to help address the ongoing
>>> and growing need for helping Mifos/Fineract scale and sustain high load
>>> environments.
>>>
>>> This really represent an area where we can demonstrate that together as
>>> Open Source Community we are much stronger than each individual or partner
>>> trying to tackle this alone.
>>>
>>> This would be for both Fineract and Fineract CN.
>>>
>>> *Why the need for this Working Group?*
>>> The formation of a working group focused on this subject is needed as
>>> evident from various mail thread or support requests trying to optimize the
>>> system for high load/volume environments and a growing number of prospects
>>> eager to use the platform but needing more visibility into its performance
>>> and ability to meet high TPS requirements.
>>>
>>> Secondly, there are many partner-led implementations and deployments of
>>> Fineract/Mifos supporting millions of clients and it would be valuable to
>>> share that knowledge across the community.
>>>
>>> Lastly, it's been quite some time since we've had public performance
>>> testing done, the efforts led by eSolve in 2017 (see wiki page) and IBM/Conflux
>>> in 2015
>>> <https://www.ibm.com/partnerworld/page/stg_ast_sys-mifos-x-on-ibm-powerlinux-servers>both
>>> pre-date the recent change from Hibernate to OpenJPA. Kumaranath with
>>> support from Avik of Fynarfin worked on performance related issues during
>>> 2018 GSOC -
>>> https://docs.google.com/document/d/18_awblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW_-In0
>>>
>>>
>>> *What/How*
>>>
>>> 1. Create a set of reproducible tools to enable automation of this
>>> on an ongoing basis.
>>> 1. Performance testing related data sets - i.e. DB dumps, scripts
>>> to generate test data
>>> 2. Postman scripts to run load tests
>>> 3. Perhaps we can create a repo on github for this?
>>> 2. Access to Resources
>>> 1. If you have hardware or cloud environments available to
>>> conduct these load/performance tests please share.
>>> 3. Culture & Transparency - Establish this as a priority area of the
>>> community to collectively address.
>>> 4. Documentation (likely on the Fineract Wiki) - I created this page
>>> to start: https://cwiki.apache.org/confluence/x/khD3Bw
>>> 1. Share and document existing results of performance testing to
>>> date. Share the details of environments you have set up to address high
>>> load needs.
>>> 2. Document the scenarios and TPS requirements that need to be
>>> tested
>>> 3. Document typical improvements to address performance (at a
>>> configuration, database, code, level, etc.)
>>> 5. Identify issues to fix
>>> 1. Log new issues, update existing issues, and tag appropriately.
>>> 6. Share code and fixes to address performance issues
>>> 1. Make these a priority and plan out in our release roadmapping.
>>>
>>> *Next Steps*
>>>
>>> - If interested in joining the group, respond to this thread.
>>> - Begin sharing your inputs on the wiki page.
>>> - Start fixing existing issues.
>>> - Help in creating and executing on our performance testing plan.
>>>
>>> Thanks,
>>>
>>> Ed
>>>
>>> Mifos-developer mailing list
>>> mifos-developer@lists.sourceforge.net
>>> Unsubscribe or change settings at:
>>> https://lists.sourceforge.net/lists/listinfo/mifos-developer
>>
>>
--
Life is a chess game - Anonymous.
Re: [Mifos-developer] Scalability & Performance Working Group
Posted by Nazeer Hussain Shaik <na...@gmail.com>.
Hi all,
Before thinking about moving micro services in different technologies
please think about getting the required skill set people and the
complexities around those technologies.
I believe scaling the micro services horizontally with proper hardware
configurations and by using some kind of load balance service (NGINX) may
be good idea.
In my current company we have more than 80 micro services and most of them
are written in Java technologies. We deploy our services on AWS (Simulating
SAAS model).
We use kubernetes for docker orchestration. Currently we are able to scale
2 million requests. We are working on to scale to 4 million requests at
least. So I don't think Java is bottleneck here.
Please check whether the DB layer and JPA is performing as expected. I
believe you need to look into client facing API(s) instead of hitting the
direct micro services, better to use some kind of Indexers (Elastic Search,
etc..). Or aggregate the data in client facing services properly.
Please correct me if my thinking is wrong.
Regards,
Nazeer
On Thu, Nov 7, 2019 at 12:54 PM Apoorva M.K <ap...@gmail.com> wrote:
> Hi Ed,
> I'm interested in helping out in this, I have gone through the list of
> next steps. I will start working on what I can.
>
> Regards,
> Apoorva
>
> On Thu 24 Oct, 2019, 1:57 AM Ed Cable, <ed...@mifos.org> wrote:
>
>> Hi Mifos and Fineract communities,
>>
>> In follow up to my previous comments on Joseph's thread I wanted to
>> initiate a collaborative community-wide effort to help address the ongoing
>> and growing need for helping Mifos/Fineract scale and sustain high load
>> environments.
>>
>> This really represent an area where we can demonstrate that together as
>> Open Source Community we are much stronger than each individual or partner
>> trying to tackle this alone.
>>
>> This would be for both Fineract and Fineract CN.
>>
>> *Why the need for this Working Group?*
>> The formation of a working group focused on this subject is needed as
>> evident from various mail thread or support requests trying to optimize the
>> system for high load/volume environments and a growing number of prospects
>> eager to use the platform but needing more visibility into its performance
>> and ability to meet high TPS requirements.
>>
>> Secondly, there are many partner-led implementations and deployments of
>> Fineract/Mifos supporting millions of clients and it would be valuable to
>> share that knowledge across the community.
>>
>> Lastly, it's been quite some time since we've had public performance
>> testing done, the efforts led by eSolve in 2017 (see wiki page) and IBM/Conflux
>> in 2015
>> <https://www.ibm.com/partnerworld/page/stg_ast_sys-mifos-x-on-ibm-powerlinux-servers>both
>> pre-date the recent change from Hibernate to OpenJPA. Kumaranath with
>> support from Avik of Fynarfin worked on performance related issues during
>> 2018 GSOC -
>> https://docs.google.com/document/d/18_awblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW_-In0
>>
>>
>> *What/How*
>>
>> 1. Create a set of reproducible tools to enable automation of this on
>> an ongoing basis.
>> 1. Performance testing related data sets - i.e. DB dumps, scripts
>> to generate test data
>> 2. Postman scripts to run load tests
>> 3. Perhaps we can create a repo on github for this?
>> 2. Access to Resources
>> 1. If you have hardware or cloud environments available to conduct
>> these load/performance tests please share.
>> 3. Culture & Transparency - Establish this as a priority area of the
>> community to collectively address.
>> 4. Documentation (likely on the Fineract Wiki) - I created this page
>> to start: https://cwiki.apache.org/confluence/x/khD3Bw
>> 1. Share and document existing results of performance testing to
>> date. Share the details of environments you have set up to address high
>> load needs.
>> 2. Document the scenarios and TPS requirements that need to be
>> tested
>> 3. Document typical improvements to address performance (at a
>> configuration, database, code, level, etc.)
>> 5. Identify issues to fix
>> 1. Log new issues, update existing issues, and tag appropriately.
>> 6. Share code and fixes to address performance issues
>> 1. Make these a priority and plan out in our release roadmapping.
>>
>> *Next Steps*
>>
>> - If interested in joining the group, respond to this thread.
>> - Begin sharing your inputs on the wiki page.
>> - Start fixing existing issues.
>> - Help in creating and executing on our performance testing plan.
>>
>> Thanks,
>>
>> Ed
>>
>> Mifos-developer mailing list
>> mifos-developer@lists.sourceforge.net
>> Unsubscribe or change settings at:
>> https://lists.sourceforge.net/lists/listinfo/mifos-developer
>
>