You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@fineract.apache.org by Giorgio Zoppi <gi...@gmail.com> on 2020/03/21 21:55:32 UTC
Fineract-CN - ActiveMQ vs Kafka
Hello Gents,
I have a couple of questions:
1. do you think feasible to replace/mantain Kafka support instead ActiveMQ
for increasing scalability and ScyllaDB instead Cassandra? For what reason
do you use Cassandra? (with those changes i bet you can gain 10%
performance on heavy load).
2. have experience with Gradle 5 and Java 11 compiling fineract-cn?
3. Which is the bare minimum to run / refactor for fineract-cn?
BR,
Giorgio
Re: Fineract-CN - ActiveMQ vs Kafka
Posted by Giorgio Zoppi <gi...@gmail.com>.
Thanks Juhan,
i try your branch and keep going on that.
Best Regards,
Giorgio
Re: Fineract-CN - ActiveMQ vs Kafka
Posted by Juhan Aasaru <aa...@gmail.com>.
We don't have to be backwards compatible with upgrade to Spring Boot 2 and
Java 11.
We can just do the upgrade.
However we need to keep the system functional and for this we need to keep
our builds (travis-ci) passing.
For this we need to start with upgrading the micro service which doesn't
depend on any other micro service (this is fineract-cn-identity)
and the libraries it depends on (visible here:
https://github.com/apache/fineract-cn-identity/blob/develop/shared.gradle#L22
)
I have created branch spring_boot_2 to each of the micro service and to
each library.
So basically if you want to upgrade identity to Spring Boot 2 (+ Java 11)
you have to send PR here:
https://github.com/apache/fineract-cn-identity/tree/spring_boot_2
And if you need to update any dependent libraries (anubis for example) you
need to get the PR approved to this branch:
https://github.com/apache/fineract-cn-anubis/tree/spring_boot_2
this is built by travis-ci to a jar and fineract-cn-idneitty depends on
this jar.
Juhan
...
Kontakt Giorgio Zoppi (<gi...@gmail.com>) kirjutas kuupäeval P, 22.
märts 2020 kell 14:10:
>
> Ok,
> the discussion is going to be interesting now:
> 1. We need a broker mechanism that will scale for and reduce the latency.
> We could support both just using a config parameter and let the user decide.
> 2. +1 DynamoDB .
>
> Thanks for the info:
> Different members of the community have upgraded but nobody has bothered
> to give it back to the community. If you want Java 11 you need to first
> upgrade to Spring Boot 2. More info:
> https://issues.apache.org/jira/browse/FINCN-172
> Do you have a repo? Is anyone interested to move to Java 11 and Spring
> Boot 2 officially all fineract-cn (and just that)? Do we have any backward
> compatibility issues?
>
> Best Regards,
> Giorgio
>
>
Re: Fineract-CN - ActiveMQ vs Kafka
Posted by Giorgio Zoppi <gi...@gmail.com>.
Ok,
the discussion is going to be interesting now:
1. We need a broker mechanism that will scale for and reduce the latency.
We could support both just using a config parameter and let the user decide.
2. +1 DynamoDB .
Thanks for the info:
Different members of the community have upgraded but nobody has bothered to
give it back to the community. If you want Java 11 you need to first
upgrade to Spring Boot 2. More info:
https://issues.apache.org/jira/browse/FINCN-172
Do you have a repo? Is anyone interested to move to Java 11 and Spring Boot
2 officially all fineract-cn (and just that)? Do we have any backward
compatibility issues?
Best Regards,
Giorgio
Re: Fineract-CN - ActiveMQ vs Kafka
Posted by Juhan Aasaru <aa...@gmail.com>.
1. The ideal solution would separate implementation from rest of the code
so different organizations could use different technologies.
I don't see much value in replacing ActiveMQ (as it work out of the box and
requires little maintenance).
I see a huge win if it would be possible to use other alternatives besides
Cassandra.
I personally would use Amazon DynamoDB but different companies have
different technology stacks in-house that they prefer.
So the discussion on the topic should not be "how replace X with Y" but
rather "how could we support alternatives to X".
2. Different members of the community have upgraded but nobody has bothered
to give it back to the community. If you want Java 11 you need to first
upgrade to Spring Boot 2. More info:
https://issues.apache.org/jira/browse/FINCN-172
BR,
Juhan
Kontakt Giorgio Zoppi (<gi...@gmail.com>) kirjutas kuupäeval L, 21.
märts 2020 kell 23:55:
> Hello Gents,
> I have a couple of questions:
> 1. do you think feasible to replace/mantain Kafka support instead ActiveMQ
> for increasing scalability and ScyllaDB instead Cassandra? For what reason
> do you use Cassandra? (with those changes i bet you can gain 10%
> performance on heavy load).
> 2. have experience with Gradle 5 and Java 11 compiling fineract-cn?
> 3. Which is the bare minimum to run / refactor for fineract-cn?
>
> BR,
> Giorgio
>
Re: Fineract-CN - ActiveMQ vs Kafka
Posted by Juhan Aasaru <aa...@gmail.com>.
Giorgio,
Storing commands is just one aspect of all the things stored in Cassandra.
I recommend you start the bare minimum of Fineract CN:
https://cwiki.apache.org/confluence/display/FINERACT/Bare+minimum+to+run+Fineract-CN+locally
And then connect to Cassandra (I use IntelliJ IDEA for this)
and see the tables inside this database.
Then you see that Cassandra is also used for storing data.
For example it keeps a record of all roles in the system.
This class is used to get / add / change or delete a role:
https://github.com/apache/fineract-cn-identity/blob/develop/service/src/main/java/org/apache/fineract/cn/identity/internal/repository/Roles.java
As you see from the code, this data is kept in Cassandra not in Postgres or
anywhere else.
As a matter of fact - Identity keeps all its data in Cassandra while other
micro services of Fineract CN
keep some persistent data in Postgres and some in Cassandra.
Juhan
Kontakt Giorgio Zoppi (<gi...@gmail.com>) kirjutas kuupäeval P, 22.
märts 2020 kell 16:03:
> Hello Gents,
> following this statement Cassandra is primarily used for logging commands
> ( see
> https://cwiki.apache.org/confluence/display/FINERACT/CQRS+in+Fineract+CN )
> and journal entries.
> We don't need Cassandra y neither a document/column db. Just Redis (with
> Streams) is enough.
>
> BR,
> Giorgio
>
>
> El dom., 22 mar. 2020 a las 0:02, Vishwas Babu (<vi...@gmail.com>)
> escribió:
>
>> Hello Giorgio,
>>
>> >>1. do you think feasible to replace/mantain Kafka support instead
>> ActiveMQ for increasing scalability and ScyllaDB instead Cassandra? For
>> what reason do you use Cassandra? (with those changes i bet you can gain
>> 10% performance on heavy load).
>>
>> Cassandra is primarily used for logging commands ( see
>> https://cwiki.apache.org/confluence/display/FINERACT/CQRS+in+Fineract+CN )
>> and journal entries.
>>
>> +1 for Kafka
>> Tentative +1 for ScyllaDB (considering potential licensing issues with
>> ScyllaDB primarily using AGPL. Drivers, etc seem to use Apache 2.0 though).
>>
>> >> 3. Which is the bare minimum to run / refactor for fineract-cn?
>>
>> Depends on the approach taken for running fineract-cn
>> ->
>> https://cwiki.apache.org/confluence/display/FINERACT/How+To+Build+Apache+Fineract+CN needs
>> the most resources
>> -> Docker-based approaches need similar / slightly lesser (?) resources.
>> https://cwiki.apache.org/confluence/display/FINERACT/Bare+minimum+to+run+Fineract-CN+locally has
>> instructions for bringing up the minimal set of services
>> -> For development, manually bringing up the required services from the
>> command line requires the least resources. Instructions at
>> https://cwiki.apache.org/confluence/display/FINERACT/Running+Fineract+CN+Locally+on+Windows
>>
>> Regards,
>> Vishwas
>>
>>
>>
>> On Sat, 21 Mar 2020 at 14:55, Giorgio Zoppi <gi...@gmail.com>
>> wrote:
>>
>>> Hello Gents,
>>> I have a couple of questions:
>>> 1. do you think feasible to replace/mantain Kafka support instead
>>> ActiveMQ for increasing scalability and ScyllaDB instead Cassandra? For
>>> what reason do you use Cassandra? (with those changes i bet you can gain
>>> 10% performance on heavy load).
>>> 2. have experience with Gradle 5 and Java 11 compiling fineract-cn?
>>> 3. Which is the bare minimum to run / refactor for fineract-cn?
>>>
>>> BR,
>>> Giorgio
>>>
>>
>
> --
> Life is a chess game - Anonymous.
>
Re: Fineract-CN - ActiveMQ vs Kafka
Posted by Giorgio Zoppi <gi...@gmail.com>.
Hello Gents,
following this statement Cassandra is primarily used for logging commands
( see
https://cwiki.apache.org/confluence/display/FINERACT/CQRS+in+Fineract+CN )
and journal entries.
We don't need Cassandra y neither a document/column db. Just Redis (with
Streams) is enough.
BR,
Giorgio
El dom., 22 mar. 2020 a las 0:02, Vishwas Babu (<vi...@gmail.com>)
escribió:
> Hello Giorgio,
>
> >>1. do you think feasible to replace/mantain Kafka support instead
> ActiveMQ for increasing scalability and ScyllaDB instead Cassandra? For
> what reason do you use Cassandra? (with those changes i bet you can gain
> 10% performance on heavy load).
>
> Cassandra is primarily used for logging commands ( see
> https://cwiki.apache.org/confluence/display/FINERACT/CQRS+in+Fineract+CN )
> and journal entries.
>
> +1 for Kafka
> Tentative +1 for ScyllaDB (considering potential licensing issues with
> ScyllaDB primarily using AGPL. Drivers, etc seem to use Apache 2.0 though).
>
> >> 3. Which is the bare minimum to run / refactor for fineract-cn?
>
> Depends on the approach taken for running fineract-cn
> ->
> https://cwiki.apache.org/confluence/display/FINERACT/How+To+Build+Apache+Fineract+CN needs
> the most resources
> -> Docker-based approaches need similar / slightly lesser (?) resources.
> https://cwiki.apache.org/confluence/display/FINERACT/Bare+minimum+to+run+Fineract-CN+locally has
> instructions for bringing up the minimal set of services
> -> For development, manually bringing up the required services from the
> command line requires the least resources. Instructions at
> https://cwiki.apache.org/confluence/display/FINERACT/Running+Fineract+CN+Locally+on+Windows
>
> Regards,
> Vishwas
>
>
>
> On Sat, 21 Mar 2020 at 14:55, Giorgio Zoppi <gi...@gmail.com>
> wrote:
>
>> Hello Gents,
>> I have a couple of questions:
>> 1. do you think feasible to replace/mantain Kafka support instead
>> ActiveMQ for increasing scalability and ScyllaDB instead Cassandra? For
>> what reason do you use Cassandra? (with those changes i bet you can gain
>> 10% performance on heavy load).
>> 2. have experience with Gradle 5 and Java 11 compiling fineract-cn?
>> 3. Which is the bare minimum to run / refactor for fineract-cn?
>>
>> BR,
>> Giorgio
>>
>
--
Life is a chess game - Anonymous.
Re: Fineract-CN - ActiveMQ vs Kafka
Posted by Vishwas Babu <vi...@gmail.com>.
Hello Giorgio,
>>1. do you think feasible to replace/mantain Kafka support instead
ActiveMQ for increasing scalability and ScyllaDB instead Cassandra? For
what reason do you use Cassandra? (with those changes i bet you can gain
10% performance on heavy load).
Cassandra is primarily used for logging commands ( see
https://cwiki.apache.org/confluence/display/FINERACT/CQRS+in+Fineract+CN )
and journal entries.
+1 for Kafka
Tentative +1 for ScyllaDB (considering potential licensing issues with
ScyllaDB primarily using AGPL. Drivers, etc seem to use Apache 2.0 though).
>> 3. Which is the bare minimum to run / refactor for fineract-cn?
Depends on the approach taken for running fineract-cn
->
https://cwiki.apache.org/confluence/display/FINERACT/How+To+Build+Apache+Fineract+CN
needs
the most resources
-> Docker-based approaches need similar / slightly lesser (?) resources.
https://cwiki.apache.org/confluence/display/FINERACT/Bare+minimum+to+run+Fineract-CN+locally
has
instructions for bringing up the minimal set of services
-> For development, manually bringing up the required services from the
command line requires the least resources. Instructions at
https://cwiki.apache.org/confluence/display/FINERACT/Running+Fineract+CN+Locally+on+Windows
Regards,
Vishwas
On Sat, 21 Mar 2020 at 14:55, Giorgio Zoppi <gi...@gmail.com> wrote:
> Hello Gents,
> I have a couple of questions:
> 1. do you think feasible to replace/mantain Kafka support instead ActiveMQ
> for increasing scalability and ScyllaDB instead Cassandra? For what reason
> do you use Cassandra? (with those changes i bet you can gain 10%
> performance on heavy load).
> 2. have experience with Gradle 5 and Java 11 compiling fineract-cn?
> 3. Which is the bare minimum to run / refactor for fineract-cn?
>
> BR,
> Giorgio
>
Re: Fineract-CN - ActiveMQ vs Kafka
Posted by Giorgio Zoppi <gi...@gmail.com>.
Did anyone use AWS Managed Cassandra Service or Azure Cosmos DB? Although,
it's available in very limited regions (5), pricing calculation on the
surface looks to be more reasonable than deploying it on EC2 as it is based
on RUs per second which is a rate-based currency. "It abstracts the system
resources such as CPU, IOPS, and memory that are required to perform the
database operations supported". If anyone has experimented with
such read/write capacity units (RCUs and WCUs) and reserved burst capacity
with FINCN, that information will be really great.
This is a good point for migrating to Cosmos or DynamoDB.
BR,
Giorgio
El lun., 23 mar. 2020 a las 14:43, Avik Ganguly (<av...@fynarfin.io>)
escribió:
> ScyllaDB instead Cassandra
>
>
> Keeping aside licensing constraints, how is ScyllaDB's tooling support?
> Cassandra's tooling built by DataStax for OpsCenter & DevCenter has
> increasingly moved towards DataStax Enterprise over the years with
> OpsCenter removing support for Cassandra completely in recent releases.
> Plus DataStax agent or any alternative OpsCenter solution (Diamond +
> Graphite or Beats + ElasticSearch) will come with it's own considerable
> resource requirements.
>
> Did anyone use AWS Managed Cassandra Service or Azure Cosmos DB? Although,
> it's available in very limited regions (5), pricing calculation on the
> surface looks to be more reasonable than deploying it on EC2 as it is based
> on RUs per second which is a rate-based currency. "It abstracts the system
> resources such as CPU, IOPS, and memory that are required to perform the
> database operations supported". If anyone has experimented with
> such read/write capacity units (RCUs and WCUs) and reserved burst capacity
> with FINCN, that information will be really great.
>
>
> On Sun, Mar 22, 2020 at 3:25 AM Giorgio Zoppi <gi...@gmail.com>
> wrote:
>
>> Hello Gents,
>> I have a couple of questions:
>> 1. do you think feasible to replace/mantain Kafka support instead
>> ActiveMQ for increasing scalability and ScyllaDB instead Cassandra? For
>> what reason do you use Cassandra? (with those changes i bet you can gain
>> 10% performance on heavy load).
>> 2. have experience with Gradle 5 and Java 11 compiling fineract-cn?
>> 3. Which is the bare minimum to run / refactor for fineract-cn?
>>
>> BR,
>> Giorgio
>>
>
>
> --
> Disclaimer:
>
> Privileged & confidential information is contained in this message
> (including all attachments). If you are not an intended recipient of this
> message, please destroy this message immediately and kindly notify
> the sender by reply e-mail. Any unauthorized use or dissemination of this
> message in any manner whatsoever, in whole or in part, is strictly
> prohibited. This e-mail, including all attachments hereto, (i) is for
> discussion purposes only and shall not be deemed or construed to be a
> professional opinion unless expressly stated otherwise, and (ii) is not
> intended, written or sent to be used, and cannot and shall not be used, for
> any unlawful purpose. This communication, including any attachments, may
> not be free of viruses, interceptions or interference, and may not be
> compatible with your systems. You should carry out your own virus checks
> before opening any attachment to this e-mail. The sender of this e-mail and
> Fynarfin Tech Private Limited shall not be liable for any damage that you
> may sustain as a result of viruses, incompleteness of this message, a delay
> in receipt of this message or computer problems experienced.
>
>
--
Life is a chess game - Anonymous.
Re: Fineract-CN - ActiveMQ vs Kafka
Posted by Avik Ganguly <av...@fynarfin.io>.
>
> ScyllaDB instead Cassandra
Keeping aside licensing constraints, how is ScyllaDB's tooling support?
Cassandra's tooling built by DataStax for OpsCenter & DevCenter has
increasingly moved towards DataStax Enterprise over the years with
OpsCenter removing support for Cassandra completely in recent releases.
Plus DataStax agent or any alternative OpsCenter solution (Diamond +
Graphite or Beats + ElasticSearch) will come with it's own considerable
resource requirements.
Did anyone use AWS Managed Cassandra Service or Azure Cosmos DB? Although,
it's available in very limited regions (5), pricing calculation on the
surface looks to be more reasonable than deploying it on EC2 as it is based
on RUs per second which is a rate-based currency. "It abstracts the system
resources such as CPU, IOPS, and memory that are required to perform the
database operations supported". If anyone has experimented with
such read/write capacity units (RCUs and WCUs) and reserved burst capacity
with FINCN, that information will be really great.
On Sun, Mar 22, 2020 at 3:25 AM Giorgio Zoppi <gi...@gmail.com>
wrote:
> Hello Gents,
> I have a couple of questions:
> 1. do you think feasible to replace/mantain Kafka support instead ActiveMQ
> for increasing scalability and ScyllaDB instead Cassandra? For what reason
> do you use Cassandra? (with those changes i bet you can gain 10%
> performance on heavy load).
> 2. have experience with Gradle 5 and Java 11 compiling fineract-cn?
> 3. Which is the bare minimum to run / refactor for fineract-cn?
>
> BR,
> Giorgio
>
--
Disclaimer:
Privileged & confidential information is contained in this message
(including all attachments). If you are not an intended recipient of this
message, please destroy this message immediately and kindly notify
the sender by reply e-mail. Any unauthorized use or dissemination of this
message in any manner whatsoever, in whole or in part, is strictly
prohibited. This e-mail, including all attachments hereto, (i) is for
discussion purposes only and shall not be deemed or construed to be a
professional opinion unless expressly stated otherwise, and (ii) is not
intended, written or sent to be used, and cannot and shall not be used, for
any unlawful purpose. This communication, including any attachments, may
not be free of viruses, interceptions or interference, and may not be
compatible with your systems. You should carry out your own virus checks
before opening any attachment to this e-mail. The sender of this e-mail and
Fynarfin Tech Private Limited shall not be liable for any damage that you
may sustain as a result of viruses, incompleteness of this message, a delay
in receipt of this message or computer problems experienced.