You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@fineract.apache.org by Ebenezer Graham <eg...@alustudent.com> on 2019/06/04 12:52:57 UTC
Re: Would it be interesting to use Apache "Trafodion" as an option
for persistence layer?
Hi Eric,
Thanks for this interesting and insightful proposal.
Please expound on how to use Trafodion in component tests and CI/CD.
If you noticed, all the services have a component-test package aside from
the unit tests. This is used to test the service during development as well
as for CI/CD - the data from the component tests is persisted to an
embedded Cassandra and embedded MariaDB/PostgreSQL helping to mimic a
similar CQRS pattern as will be used in production.
How will this work when using Trafodion? Does Trafodion have a light-weight
version for this use case? If not, what is a typical approach for this?
I have looked at the specs for a basic Trafodion setup and its quite
resource intensive.
Looking forward to your response
*At your service,*
*Ebenezer Graham*
On Fri, 31 May 2019 at 02:45, Eric Owhadi <er...@esgyn.com> wrote:
> Hi James,
>
> Thanks for taking the time to answer,
>
> Please find bellow inline answers to your questions.
>
>
>
> What would be the impact on current efforts to remove current dependencies
> on non Apache license compatible data persistent layers/libraries?
>
> >>> I am not yet familiar with the code, so can’t comment on this. What I
> can say is that Trafodion uses a standard jdbc driver, and other than
> dealing with minor SQL dialect differences that may exist (like any popular
> RDBMS dialect minor differences), the adaptation to Trafodion should be
> minimal (depending on how the code is structured – understand if the SQL
> statement are well separated vs have to hunt for them in many places in the
> code).
>
> Having said that, the statement is valid for just replacing RDBMS calls to
> trafodion. Then replacing Cassandra calls to trafodion should also be easy.
> However, if we want to replace some of current CQRS commands from Async to
> Sync, the code change will be more involved, as I assume the whole thing
> about doing “light check” before performing the async command, and the
> un-necessary error handling involved at the use case level (what happen if
> an async command fail, and UI thought it went OK) should be optionally
> removed. Instead classic RDBMS transaction commit/rollback would be done.
>
>
>
> Once implemented, is this more, less, same to maintain and add
> functionality? Would it require a dev to map the data logical level (ORM)
> more intensively? Is there a different skill set needed with this than
> current concept and are those skills generally available?
>
> >>> general DB skill. Easier than current solution, as standard SQL is
> easier to find than standard SQL + Cassandra special.
>
>
>
> Is the trajectory of this technology vs noSql clear?
>
> >>> Yes. It’s purpose is to add back everything that was given up (ACID)
> from classic RDBMS to achieve horizontal scalability. Now you have a full
> classic RDBMS, but capable of handling huge volume, with scale out share
> nothing architecture.
>
>
>
> You've made a good case, what's the counter?
>
> >>> The counter argument would be maturity. MySQL, Postgres or MariaDB,
> given their wide adoption, benefit from more battle testing than Trafodion.
> Having said that, and given the recent commitment to core banking, EsgynDB
> will be battle tested for core Banking, while the other DB are not
> typically seen in the “classic” Core Banking sector. By the time
> fineract-cn will be feature complete, I think the maturity argument will
> have seriously faded out.
>
>
>
> What about Apache Ignite?
>
> >>> Apache Ignite is a different beast. Classified as “in memory DB”. I
> have no experience with it, but can say that what is needed is a
> persistence layer, not an in memory db to speed up analytical queries.
> Therefore I have hard time seeing a fit with Ignite.
>
>
>
> Is there an independent review of Trafodion?
>
> >>> Not that I know of.
>
>
>
> Is the open source version useable or does this require use of the EsgynDB
> in production? Are other providers using the tech?
>
> >>> I would not use Trafodion in production, given I would never go in
> production without support contract on the database. Currently only Esgyn
> Corporation provide a Trafodion based production ready product.
>
>
>
>
>
> If the project devs can come to some conclusion on this, what's the level
> of effort? How should we do think about the size of this effort. & do you
> think... is there someone available to work on this project, perhaps from
> one of those banks?
>
> >>> On my free cycles, I will study fineract-cn code and analyze effort /
> prototype something.
>
>
>
> Best regards,
>
> Eric Owhadi
>
>
>
> *From:* James Dailey <ja...@gmail.com>
> *Sent:* Thursday, May 30, 2019 4:26 PM
> *To:* Dev <de...@fineract.apache.org>
> *Cc:* Eric Owhadi <er...@esgyn.com>
> *Subject:* Re: Would it be interesting to use Apache "Trafodion" as an
> option for persistence layer?
>
>
>
> External
>
> Eric,
>
>
>
> First, thank you very much.
>
>
>
> This is a really interesting concept and proposal for a shift in
> architecture. I have some questions:
>
>
>
> What would be the impact on current efforts to remove current dependencies
> on non Apache license compatible data persistent layers/libraries?
>
>
>
> Once implemented, is this more, less, same to maintain and add
> functionality? Would it require a dev to map the data logical level (ORM)
> more intensively? Is there a different skill set needed with this than
> current concept and are those skills generally available?
>
>
>
> Is the trajectory of this technology vs noSql clear? What about Apache
> Ignite? Is there an independent review of Trafodion? You've made a good
> case, what's the counter?
>
>
>
> Is the open source version useable or does this require use of the EsgynDB
> in production? Are other providers using the tech?
>
>
>
> If the project devs can come to some conclusion on this, what's the level
> of effort? How should we do think about the size of this effort. & do you
> think... is there someone available to work on this project, perhaps from
> one of those banks?
>
>
>
> Thanks,
>
> Jdailey67
>
>
>
>
>
> On Thu, May 30, 2019, 1:48 AM Eric Owhadi <er...@sbcglobal.net>
> wrote:
>
> Hello Fineracteers,
>
> I discovered Fineract and specially Fineract-cn last week, and am very
> impressed by the future potential of the project, both to deliver on its
> original social mission but also to revolutionize commercial FinTech sector
> the same way the big data wave caught up when Hadoop was open sourced.
>
> The architecture [Cloud Native/micro-services] is at the forefront of what
> a modern application should look like, and the decision to start from
> scratch and don’t carry the burden of legacy was a bold but very smart move
> imho.
>
> There is however, one area that might be interesting to challenge: the
> CQRS and the dual storage engine needed to provide capability for async
> command execution (and its draw back on error handling from front end) and
> sync query accessing the traditional DBMS with its scalability shortfall.
>
>
>
> The current implementation looks optimal when it was picked (I assume when
> fineract-cn was still mifos io), because there was no open source “NewSQL”
> solution available. Using classic RDBMS with known scalability limitation,
> coupled with NoSQL eventual consistency, no ACID, no multicolumn/table
> transaction, no join or grouping, given up to gain stellar horizontal
> scalability was a smart move.
>
>
>
> However, in 2017, Apache released Trafodion as Top Level Project. It could
> replace both Postgress/MariaDb/MySQL and Cassandra. It is an MPP RDBMS
> built on top of Hadoop Hbase (the Hadoop equivalent of Cassandra). It is
> fully ACID, supports multicolumn/multi table transactions, joins, group by,
> is fully horizontally scalable and support all the bells and whistle you
> would expect from a classic RDBMS. It can be classified as NewSQL,
> basically claiming back all the things given up by NoSql in exchange for
> scalability. In addition, the company backing it up (Esgyn Corp) have
> recently announced that its commercial product (EsgynDB, derived from
> Trafodion) is being used for Core Banking in 2 major Banks in China,
> struggling with the ever growing mobile payment volume that traditional
> scale up databases cannot keep up with. In addition Esgyn provides a DB as
> a service offering, cloud hosted fully managed (Esgyn Strato), giving the
> option to potential fineract-cn hosting company to “outsource” the
> persistence layer management and focus on the rest.
>
>
>
> Wouldn’t it be an interesting idea to hook Trafodion to Fineract-cn? I can
> think of the following advantages:
>
> - Simplify persistence layer ecosystem from 2 DB to 1 -> back to
> classic RDBMS but with full scale out capability MPP style.
> - Still Apache license compatible
> - Persistence layer ready to support challenging analytical workload
> (reporting use cases? - thanks for reclaiming Join/group-by at scale).
> - Remove the reticence of decision maker to use NoSQL in Core Banking
> - Performance over Cassandra [on paper, even if Trafodion was used
> only as a drop in replacement to Cassandra for async commands, the use of
> Trafodion “Align Format” feature should demonstrate higher performance for
> journaling, as Trafodion would write 1 BigTable cell per row, while
> Cassandra or Native Hbase would write 1 X nb Column cells per row].
> - With a solid company already showing its commitment to core banking
> backing the Apache project
>
>
>
> Hope this makes sense?
>
> Eric Owhadi,
>
> https://www.linkedin.com/in/eric-owhadi-3173058/
>
> Principal Engineer at Esgyn Corporation.
>
> message resent from personal email, the first tentative did not make it...
>
>
RE: Would it be interesting to use Apache "Trafodion" as an option
for persistence layer?
Posted by Eric Owhadi <er...@esgyn.com>.
Hi Markus,
Thanks for your message and for sharing your experience with a large retail German bank.
You may be familiar with this:
https://martinfowler.com/bliki/CQRS.html
The author explains the pro/cons of CQRS design pattern/ when to use vs when to avoid.
Also, from fineract documentation, the following has been identified
https://cwiki.apache.org/confluence/display/FINERACT/CQRS+in+Fineract+CN
“The advantage to this approach is that it's really fast. The drawback is that some kinds of errors can only be detected while the command handler is executing and this class of errors will not be surfaced appropriately to the user. The basic sanity checks are intended to ameliorate this.”
You can find from feedback published on fineract wiki
https://cwiki.apache.org/confluence/display/FINERACT/Hosting+Experiences+from+Partners#HostingExperiencesfromPartners-Mines.io
“Concern about using Cassandra for journal: noSQL is not a typical choice for strongly consistent use case”
And the message bellow from Victor Romero having his company investigate a NewSQL alternative to the NoSQL + MariaDB/MySQL/Postgres
There is an operational cost associated to running a database (backup/restore/upgrade/patch/admin etc). So the least amount of DB instances, the cheaper the TCO of the solution.
Last point that comes to mind: having the RDBMS leg horizontally scalable allows the “Query” side of CQRS to be also horizontally scalable.
Hope this makes sense?
Best regards,
Eric
From: Markus Geiss <ma...@apache.org>
Sent: Wednesday, June 5, 2019 1:36 AM
To: Dev <de...@fineract.apache.org>
Subject: Re: Would it be interesting to use Apache "Trafodion" as an option for persistence layer?
Hey all,
trust this finds you well. (;
The whole architecture of CN is build around a clean separation of concerns, e.g. commands vs queries.
Based on the tools we are using (e.g. Spring data, Spring messaging) and have built on top (e.g. Command gateway) on a day to day base writing business value code will not interfere with technical details or writing SQL statements.
I'm not sure about what kind of complexity Eric is talking about, and would be happy to hear about these concerns.
PS: We are talking to large retail banks here in Germany, and neither from an internal objection nor from a regulatory point of view NoSQL is an issue.
Markus
On Wed, Jun 5, 2019 at 7:33 AM Victor Romero <vi...@fintecheando.mx>> wrote:
Hi Eric,
You have bring a good topic on the table, the use of NewSQL in Core Banking. I think that is important to put in the center of the discussion concept of "NewSQL" instead of naming a solution before researching, evaluating and sharing experiences.
Recently we have done some research for supporting a request from an enterprise user, because the customer have told us the concerns that the institution has about MariaDB (SQL) and Cassandra (NoSQL) used in Fineract.
Then we did the leg-work comparing NewSQL options: Google Spanner (Not an option because data regulation issues), CockroachDB (Time consuming for changing the code), VoltDB (Time consuming for changing the code) & Trafodion (Time consuming for changing the code) and TiDB (now time to explain very briefly)
In our case we have decided to do Proof of Concepts using TiDB (NewSQL) because of these drivers:
- Scalability
- High Availability
- ACID
- Ease of use
---- It is compatible with the MySQL ecosystem tools
---- Reuse the knowledge of our data engineering team
---- Reduce the time to port/close the SQL gaps ( https://pingcap.com/docs/dev/reference/mysql-compatibility/ )
- Existing customer experience published by the "vendor" of the DB
---- We have contacted companies to gather information about the use/migraction experience of TiDB (https://github.com/pingcap/docs/blob/master/adopters.md and https://www.pingcap.com/success-stories/tidb-in-shopee/ )
We will share the experience by the end of this month about the use if Fineract CN and TiDB for each driver.
Just to give more information about the DBs, I will take a table and a image from the paper "TOP NEWSQL DATABASES AND FEATURES CLASSIFICATION" - International Journal of Database Management Systems ( IJDMS ) Vol.10, No.2, April 201, for summarizing the concepts about SQL, NoSQL and NewSQL
[cid:image001.png@01D51BA0.5ED91520]
[cid:image002.png@01D51BA0.5ED91520]
By the way, the document has evaluated Trafodion :) and the authors give their opinions about it, it is worth reading.
Regards
Victor
El 04/06/19 a las 20:12, Eric Owhadi escribió:
Hi Graham,
Thanks you for your comment and question.
Trafodion does have a light version that we use for dev. We call it local_hadoop, and it fits in single dev boxes. This will perfectly fit this use cases,
Hope this helps,
Eric
From: Ebenezer Graham <eg...@alustudent.com>
Sent: Tuesday, June 4, 2019 7:53 AM
To: Dev <de...@fineract.apache.org>
Subject: Re: Would it be interesting to use Apache "Trafodion" as an option for persistence layer?
External
Hi Eric,
Thanks for this interesting and insightful proposal.
Please expound on how to use Trafodion in component tests and CI/CD.
If you noticed, all the services have a component-test package aside from the unit tests. This is used to test the service during development as well as for CI/CD - the data from the component tests is persisted to an embedded Cassandra and embedded MariaDB/PostgreSQL helping to mimic a similar CQRS pattern as will be used in production.
How will this work when using Trafodion? Does Trafodion have a light-weight version for this use case? If not, what is a typical approach for this?
I have looked at the specs for a basic Trafodion setup and its quite resource intensive.
Looking forward to your response
At your service,
Ebenezer Graham
On Fri, 31 May 2019 at 02:45, Eric Owhadi <er...@esgyn.com>> wrote:
Hi James,
Thanks for taking the time to answer,
Please find bellow inline answers to your questions.
What would be the impact on current efforts to remove current dependencies on non Apache license compatible data persistent layers/libraries?
>>> I am not yet familiar with the code, so can’t comment on this. What I can say is that Trafodion uses a standard jdbc driver, and other than dealing with minor SQL dialect differences that may exist (like any popular RDBMS dialect minor differences), the adaptation to Trafodion should be minimal (depending on how the code is structured – understand if the SQL statement are well separated vs have to hunt for them in many places in the code).
Having said that, the statement is valid for just replacing RDBMS calls to trafodion. Then replacing Cassandra calls to trafodion should also be easy. However, if we want to replace some of current CQRS commands from Async to Sync, the code change will be more involved, as I assume the whole thing about doing “light check” before performing the async command, and the un-necessary error handling involved at the use case level (what happen if an async command fail, and UI thought it went OK) should be optionally removed. Instead classic RDBMS transaction commit/rollback would be done.
Once implemented, is this more, less, same to maintain and add functionality? Would it require a dev to map the data logical level (ORM) more intensively? Is there a different skill set needed with this than current concept and are those skills generally available?
>>> general DB skill. Easier than current solution, as standard SQL is easier to find than standard SQL + Cassandra special.
Is the trajectory of this technology vs noSql clear?
>>> Yes. It’s purpose is to add back everything that was given up (ACID) from classic RDBMS to achieve horizontal scalability. Now you have a full classic RDBMS, but capable of handling huge volume, with scale out share nothing architecture.
You've made a good case, what's the counter?
>>> The counter argument would be maturity. MySQL, Postgres or MariaDB, given their wide adoption, benefit from more battle testing than Trafodion. Having said that, and given the recent commitment to core banking, EsgynDB will be battle tested for core Banking, while the other DB are not typically seen in the “classic” Core Banking sector. By the time fineract-cn will be feature complete, I think the maturity argument will have seriously faded out.
What about Apache Ignite?
>>> Apache Ignite is a different beast. Classified as “in memory DB”. I have no experience with it, but can say that what is needed is a persistence layer, not an in memory db to speed up analytical queries. Therefore I have hard time seeing a fit with Ignite.
Is there an independent review of Trafodion?
>>> Not that I know of.
Is the open source version useable or does this require use of the EsgynDB in production? Are other providers using the tech?
>>> I would not use Trafodion in production, given I would never go in production without support contract on the database. Currently only Esgyn Corporation provide a Trafodion based production ready product.
If the project devs can come to some conclusion on this, what's the level of effort? How should we do think about the size of this effort. & do you think... is there someone available to work on this project, perhaps from one of those banks?
>>> On my free cycles, I will study fineract-cn code and analyze effort / prototype something.
Best regards,
Eric Owhadi
From: James Dailey <ja...@gmail.com>>
Sent: Thursday, May 30, 2019 4:26 PM
To: Dev <de...@fineract.apache.org>>
Cc: Eric Owhadi <er...@esgyn.com>>
Subject: Re: Would it be interesting to use Apache "Trafodion" as an option for persistence layer?
External
Eric,
First, thank you very much.
This is a really interesting concept and proposal for a shift in architecture. I have some questions:
What would be the impact on current efforts to remove current dependencies on non Apache license compatible data persistent layers/libraries?
Once implemented, is this more, less, same to maintain and add functionality? Would it require a dev to map the data logical level (ORM) more intensively? Is there a different skill set needed with this than current concept and are those skills generally available?
Is the trajectory of this technology vs noSql clear? What about Apache Ignite? Is there an independent review of Trafodion? You've made a good case, what's the counter?
Is the open source version useable or does this require use of the EsgynDB in production? Are other providers using the tech?
If the project devs can come to some conclusion on this, what's the level of effort? How should we do think about the size of this effort. & do you think... is there someone available to work on this project, perhaps from one of those banks?
Thanks,
Jdailey67
On Thu, May 30, 2019, 1:48 AM Eric Owhadi <er...@sbcglobal.net>> wrote:
Hello Fineracteers,
I discovered Fineract and specially Fineract-cn last week, and am very impressed by the future potential of the project, both to deliver on its original social mission but also to revolutionize commercial FinTech sector the same way the big data wave caught up when Hadoop was open sourced.
The architecture [Cloud Native/micro-services] is at the forefront of what a modern application should look like, and the decision to start from scratch and don’t carry the burden of legacy was a bold but very smart move imho.
There is however, one area that might be interesting to challenge: the CQRS and the dual storage engine needed to provide capability for async command execution (and its draw back on error handling from front end) and sync query accessing the traditional DBMS with its scalability shortfall.
The current implementation looks optimal when it was picked (I assume when fineract-cn was still mifos io), because there was no open source “NewSQL” solution available. Using classic RDBMS with known scalability limitation, coupled with NoSQL eventual consistency, no ACID, no multicolumn/table transaction, no join or grouping, given up to gain stellar horizontal scalability was a smart move.
However, in 2017, Apache released Trafodion as Top Level Project. It could replace both Postgress/MariaDb/MySQL and Cassandra. It is an MPP RDBMS built on top of Hadoop Hbase (the Hadoop equivalent of Cassandra). It is fully ACID, supports multicolumn/multi table transactions, joins, group by, is fully horizontally scalable and support all the bells and whistle you would expect from a classic RDBMS. It can be classified as NewSQL, basically claiming back all the things given up by NoSql in exchange for scalability. In addition, the company backing it up (Esgyn Corp) have recently announced that its commercial product (EsgynDB, derived from Trafodion) is being used for Core Banking in 2 major Banks in China, struggling with the ever growing mobile payment volume that traditional scale up databases cannot keep up with. In addition Esgyn provides a DB as a service offering, cloud hosted fully managed (Esgyn Strato), giving the option to potential fineract-cn hosting company to “outsource” the persistence layer management and focus on the rest.
Wouldn’t it be an interesting idea to hook Trafodion to Fineract-cn? I can think of the following advantages:
* Simplify persistence layer ecosystem from 2 DB to 1 -> back to classic RDBMS but with full scale out capability MPP style.
* Still Apache license compatible
* Persistence layer ready to support challenging analytical workload (reporting use cases? - thanks for reclaiming Join/group-by at scale).
* Remove the reticence of decision maker to use NoSQL in Core Banking
* Performance over Cassandra [on paper, even if Trafodion was used only as a drop in replacement to Cassandra for async commands, the use of Trafodion “Align Format” feature should demonstrate higher performance for journaling, as Trafodion would write 1 BigTable cell per row, while Cassandra or Native Hbase would write 1 X nb Column cells per row].
* With a solid company already showing its commitment to core banking backing the Apache project
Hope this makes sense?
Eric Owhadi,
https://www.linkedin.com/in/eric-owhadi-3173058/
Principal Engineer at Esgyn Corporation.
message resent from personal email, the first tentative did not make it...
Re: Would it be interesting to use Apache "Trafodion" as an option
for persistence layer?
Posted by Markus Geiss <ma...@apache.org>.
Hey all,
trust this finds you well. (;
The whole architecture of CN is build around a clean separation of
concerns, e.g. commands vs queries.
Based on the tools we are using (e.g. Spring data, Spring messaging) and
have built on top (e.g. Command gateway) on a day to day base writing
business value code will not interfere with technical details or writing
SQL statements.
I'm not sure about what kind of complexity Eric is talking about, and would
be happy to hear about these concerns.
PS: We are talking to large retail banks here in Germany, and neither from
an internal objection nor from a regulatory point of view NoSQL is an issue.
Markus
On Wed, Jun 5, 2019 at 7:33 AM Victor Romero <vi...@fintecheando.mx>
wrote:
> Hi Eric,
>
> You have bring a good topic on the table, the use of NewSQL in Core
> Banking. I think that is important to put in the center of the discussion
> concept of "NewSQL" instead of naming a solution before researching,
> evaluating and sharing experiences.
>
> Recently we have done some research for supporting a request from an
> enterprise user, because the customer have told us the concerns that the
> institution has about MariaDB (SQL) and Cassandra (NoSQL) used in Fineract.
> Then we did the leg-work comparing NewSQL options: Google Spanner (Not an
> option because data regulation issues), CockroachDB (Time consuming for
> changing the code), VoltDB (Time consuming for changing the code) &
> Trafodion (Time consuming for changing the code) and TiDB (now time to
> explain very briefly)
>
> In our case we have decided to do Proof of Concepts using TiDB (NewSQL)
> because of these drivers:
>
> - Scalability
>
> - High Availability
>
> - ACID
>
> - Ease of use
>
> ---- It is compatible with the MySQL ecosystem tools
>
> ---- Reuse the knowledge of our data engineering team
>
> ---- Reduce the time to port/close the SQL gaps (
> https://pingcap.com/docs/dev/reference/mysql-compatibility/ )
>
> - Existing customer experience published by the "vendor" of the DB
>
> ---- We have contacted companies to gather information about the
> use/migraction experience of TiDB (
> https://github.com/pingcap/docs/blob/master/adopters.md and
> https://www.pingcap.com/success-stories/tidb-in-shopee/ )
>
> *We will share the experience by the end of this month about the use if
> Fineract CN and TiDB for each driver.*
>
> Just to give more information about the DBs, I will take a table and a
> image from the paper "TOP NEWSQL DATABASES AND FEATURES CLASSIFICATION" -
> International Journal of Database Management Systems ( IJDMS ) Vol.10,
> No.2, April 201, for summarizing the concepts about SQL, NoSQL and NewSQL
>
>
> By the way, the document has evaluated Trafodion :) and the authors give
> their opinions about it, it is worth reading.
>
> Regards
>
> Victor
> El 04/06/19 a las 20:12, Eric Owhadi escribió:
>
> Hi Graham,
>
> Thanks you for your comment and question.
>
> Trafodion does have a light version that we use for dev. We call it
> local_hadoop, and it fits in single dev boxes. This will perfectly fit this
> use cases,
>
> Hope this helps,
> Eric
>
>
>
> *From:* Ebenezer Graham <eg...@alustudent.com>
> <eg...@alustudent.com>
> *Sent:* Tuesday, June 4, 2019 7:53 AM
> *To:* Dev <de...@fineract.apache.org> <de...@fineract.apache.org>
> *Subject:* Re: Would it be interesting to use Apache "Trafodion" as an
> option for persistence layer?
>
>
>
> External
>
> Hi Eric,
>
>
>
> Thanks for this interesting and insightful proposal.
>
>
>
> Please expound on how to use Trafodion in component tests and CI/CD.
>
> If you noticed, all the services have a component-test package aside from
> the unit tests. This is used to test the service during development as
> well as for CI/CD - the data from the component tests is persisted to an
> embedded Cassandra and embedded MariaDB/PostgreSQL helping to mimic a
> similar CQRS pattern as will be used in production.
>
>
>
> How will this work when using Trafodion? Does Trafodion have a
> light-weight version for this use case? If not, what is a typical approach
> for this?
>
>
>
> I have looked at the specs for a basic Trafodion setup and its quite
> resource intensive.
>
>
>
> Looking forward to your response
>
>
>
> *At your service**,*
>
> *Ebenezer Graham*
>
>
>
> On Fri, 31 May 2019 at 02:45, Eric Owhadi <er...@esgyn.com> wrote:
>
> Hi James,
>
> Thanks for taking the time to answer,
>
> Please find bellow inline answers to your questions.
>
>
>
> What would be the impact on current efforts to remove current dependencies
> on non Apache license compatible data persistent layers/libraries?
>
> >>> I am not yet familiar with the code, so can’t comment on this. What I
> can say is that Trafodion uses a standard jdbc driver, and other than
> dealing with minor SQL dialect differences that may exist (like any popular
> RDBMS dialect minor differences), the adaptation to Trafodion should be
> minimal (depending on how the code is structured – understand if the SQL
> statement are well separated vs have to hunt for them in many places in the
> code).
>
> Having said that, the statement is valid for just replacing RDBMS calls to
> trafodion. Then replacing Cassandra calls to trafodion should also be easy.
> However, if we want to replace some of current CQRS commands from Async to
> Sync, the code change will be more involved, as I assume the whole thing
> about doing “light check” before performing the async command, and the
> un-necessary error handling involved at the use case level (what happen if
> an async command fail, and UI thought it went OK) should be optionally
> removed. Instead classic RDBMS transaction commit/rollback would be done.
>
>
>
> Once implemented, is this more, less, same to maintain and add
> functionality? Would it require a dev to map the data logical level (ORM)
> more intensively? Is there a different skill set needed with this than
> current concept and are those skills generally available?
>
> >>> general DB skill. Easier than current solution, as standard SQL is
> easier to find than standard SQL + Cassandra special.
>
>
>
> Is the trajectory of this technology vs noSql clear?
>
> >>> Yes. It’s purpose is to add back everything that was given up (ACID)
> from classic RDBMS to achieve horizontal scalability. Now you have a full
> classic RDBMS, but capable of handling huge volume, with scale out share
> nothing architecture.
>
>
>
> You've made a good case, what's the counter?
>
> >>> The counter argument would be maturity. MySQL, Postgres or MariaDB,
> given their wide adoption, benefit from more battle testing than Trafodion.
> Having said that, and given the recent commitment to core banking, EsgynDB
> will be battle tested for core Banking, while the other DB are not
> typically seen in the “classic” Core Banking sector. By the time
> fineract-cn will be feature complete, I think the maturity argument will
> have seriously faded out.
>
>
>
> What about Apache Ignite?
>
> >>> Apache Ignite is a different beast. Classified as “in memory DB”. I
> have no experience with it, but can say that what is needed is a
> persistence layer, not an in memory db to speed up analytical queries.
> Therefore I have hard time seeing a fit with Ignite.
>
>
>
> Is there an independent review of Trafodion?
>
> >>> Not that I know of.
>
>
>
> Is the open source version useable or does this require use of the EsgynDB
> in production? Are other providers using the tech?
>
> >>> I would not use Trafodion in production, given I would never go in
> production without support contract on the database. Currently only Esgyn
> Corporation provide a Trafodion based production ready product.
>
>
>
>
>
> If the project devs can come to some conclusion on this, what's the level
> of effort? How should we do think about the size of this effort. & do you
> think... is there someone available to work on this project, perhaps from
> one of those banks?
>
> >>> On my free cycles, I will study fineract-cn code and analyze effort /
> prototype something.
>
>
>
> Best regards,
>
> Eric Owhadi
>
>
>
> *From:* James Dailey <ja...@gmail.com>
> *Sent:* Thursday, May 30, 2019 4:26 PM
> *To:* Dev <de...@fineract.apache.org>
> *Cc:* Eric Owhadi <er...@esgyn.com>
> *Subject:* Re: Would it be interesting to use Apache "Trafodion" as an
> option for persistence layer?
>
>
>
> External
>
> Eric,
>
>
>
> First, thank you very much.
>
>
>
> This is a really interesting concept and proposal for a shift in
> architecture. I have some questions:
>
>
>
> What would be the impact on current efforts to remove current dependencies
> on non Apache license compatible data persistent layers/libraries?
>
>
>
> Once implemented, is this more, less, same to maintain and add
> functionality? Would it require a dev to map the data logical level (ORM)
> more intensively? Is there a different skill set needed with this than
> current concept and are those skills generally available?
>
>
>
> Is the trajectory of this technology vs noSql clear? What about Apache
> Ignite? Is there an independent review of Trafodion? You've made a good
> case, what's the counter?
>
>
>
> Is the open source version useable or does this require use of the EsgynDB
> in production? Are other providers using the tech?
>
>
>
> If the project devs can come to some conclusion on this, what's the level
> of effort? How should we do think about the size of this effort. & do you
> think... is there someone available to work on this project, perhaps from
> one of those banks?
>
>
>
> Thanks,
>
> Jdailey67
>
>
>
>
>
> On Thu, May 30, 2019, 1:48 AM Eric Owhadi <er...@sbcglobal.net>
> wrote:
>
> Hello Fineracteers,
>
> I discovered Fineract and specially Fineract-cn last week, and am very
> impressed by the future potential of the project, both to deliver on its
> original social mission but also to revolutionize commercial FinTech sector
> the same way the big data wave caught up when Hadoop was open sourced.
>
> The architecture [Cloud Native/micro-services] is at the forefront of what
> a modern application should look like, and the decision to start from
> scratch and don’t carry the burden of legacy was a bold but very smart move
> imho.
>
> There is however, one area that might be interesting to challenge: the
> CQRS and the dual storage engine needed to provide capability for async
> command execution (and its draw back on error handling from front end) and
> sync query accessing the traditional DBMS with its scalability shortfall.
>
>
>
> The current implementation looks optimal when it was picked (I assume when
> fineract-cn was still mifos io), because there was no open source “NewSQL”
> solution available. Using classic RDBMS with known scalability limitation,
> coupled with NoSQL eventual consistency, no ACID, no multicolumn/table
> transaction, no join or grouping, given up to gain stellar horizontal
> scalability was a smart move.
>
>
>
> However, in 2017, Apache released Trafodion as Top Level Project. It could
> replace both Postgress/MariaDb/MySQL and Cassandra. It is an MPP RDBMS
> built on top of Hadoop Hbase (the Hadoop equivalent of Cassandra). It is
> fully ACID, supports multicolumn/multi table transactions, joins, group by,
> is fully horizontally scalable and support all the bells and whistle you
> would expect from a classic RDBMS. It can be classified as NewSQL,
> basically claiming back all the things given up by NoSql in exchange for
> scalability. In addition, the company backing it up (Esgyn Corp) have
> recently announced that its commercial product (EsgynDB, derived from
> Trafodion) is being used for Core Banking in 2 major Banks in China,
> struggling with the ever growing mobile payment volume that traditional
> scale up databases cannot keep up with. In addition Esgyn provides a DB as
> a service offering, cloud hosted fully managed (Esgyn Strato), giving the
> option to potential fineract-cn hosting company to “outsource” the
> persistence layer management and focus on the rest.
>
>
>
> Wouldn’t it be an interesting idea to hook Trafodion to Fineract-cn? I can
> think of the following advantages:
>
> - Simplify persistence layer ecosystem from 2 DB to 1 -> back to
> classic RDBMS but with full scale out capability MPP style.
> - Still Apache license compatible
> - Persistence layer ready to support challenging analytical workload
> (reporting use cases? - thanks for reclaiming Join/group-by at scale).
> - Remove the reticence of decision maker to use NoSQL in Core Banking
> - Performance over Cassandra [on paper, even if Trafodion was used
> only as a drop in replacement to Cassandra for async commands, the use of
> Trafodion “Align Format” feature should demonstrate higher performance for
> journaling, as Trafodion would write 1 BigTable cell per row, while
> Cassandra or Native Hbase would write 1 X nb Column cells per row].
> - With a solid company already showing its commitment to core banking
> backing the Apache project
>
>
>
> Hope this makes sense?
>
> Eric Owhadi,
>
> https://www.linkedin.com/in/eric-owhadi-3173058/
>
> Principal Engineer at Esgyn Corporation.
>
> message resent from personal email, the first tentative did not make it...
>
>
Re: Would it be interesting to use Apache "Trafodion" as an option
for persistence layer?
Posted by Victor Romero <vi...@fintecheando.mx>.
Hi Eric,
You have bring a good topic on the table, the use of NewSQL in Core
Banking. I think that is important to put in the center of the
discussion concept of "NewSQL" instead of naming a solution before
researching, evaluating and sharing experiences.
Recently we have done some research for supporting a request from an
enterprise user, because the customer have told us the concerns that the
institution has about MariaDB (SQL) and Cassandra (NoSQL) used in Fineract.
Then we did the leg-work comparing NewSQL options: Google Spanner (Not
an option because data regulation issues), CockroachDB (Time consuming
for changing the code), VoltDB (Time consuming for changing the code) &
Trafodion (Time consuming for changing the code) and TiDB (now time to
explain very briefly)
In our case we have decided to do Proof of Concepts using TiDB (NewSQL)
because of these drivers:
- Scalability
- High Availability
- ACID
- Ease of use
---- It is compatible with the MySQL ecosystem tools
---- Reuse the knowledge of our data engineering team
---- Reduce the time to port/close the SQL gaps (
https://pingcap.com/docs/dev/reference/mysql-compatibility/ )
- Existing customer experience published by the "vendor" of the DB
---- We have contacted companies to gather information about the
use/migraction experience of TiDB
(https://github.com/pingcap/docs/blob/master/adopters.md and
https://www.pingcap.com/success-stories/tidb-in-shopee/ )
*We will share the experience by the end of this month about the use if
Fineract CN and TiDB for each driver.*
Just to give more information about the DBs, I will take a table and a
image from the paper "TOP NEWSQL DATABASES AND FEATURES CLASSIFICATION"
- International Journal of Database Management Systems ( IJDMS ) Vol.10,
No.2, April 201, for summarizing the concepts about SQL, NoSQL and NewSQL
By the way, the document has evaluated Trafodion :) and the authors give
their opinions about it, it is worth reading.
Regards
Victor
El 04/06/19 a las 20:12, Eric Owhadi escribió:
>
> Hi Graham,
>
> Thanks you for your comment and question.
>
> Trafodion does have a light version that we use for dev. We call it
> local_hadoop, and it fits in single dev boxes. This will perfectly fit
> this use cases,
>
> Hope this helps,
> Eric
>
> *From:* Ebenezer Graham <eg...@alustudent.com>
> *Sent:* Tuesday, June 4, 2019 7:53 AM
> *To:* Dev <de...@fineract.apache.org>
> *Subject:* Re: Would it be interesting to use Apache "Trafodion" as an
> option for persistence layer?
>
> External
>
> Hi Eric,
>
> Thanks for this interesting and insightful proposal.
>
> Please expound on how to use Trafodion in component tests and CI/CD.
>
> If you noticed, allthe services have a component-testpackage aside
> from the unit tests. This is used to test the serviceduring
> development as well as for CI/CD - the data from the component tests
> is persisted to an embedded Cassandra and embedded
> MariaDB/PostgreSQLhelping to mimic a similar CQRS pattern as will be
> used in production.
>
> How will this work when using Trafodion? Does Trafodion have a
> light-weight version for this use case? If not, what is a typical
> approach for this?
>
> I have looked at the specs for a basic Trafodion setup and its quite
> resource intensive.
>
> Looking forward to your response
>
> *At your service**,*
>
> *Ebenezer Graham*
>
> On Fri, 31 May 2019 at 02:45, Eric Owhadi <eric.owhadi@esgyn.com
> <ma...@esgyn.com>> wrote:
>
> Hi James,
>
> Thanks for taking the time to answer,
>
> Please find bellow inline answers to your questions.
>
> What would be the impact on current efforts to remove current
> dependencies on non Apache license compatible data persistent
> layers/libraries?
>
> >>> I am not yet familiar with the code, so can’t comment on this.
> What I can say is that Trafodion uses a standard jdbc driver, and
> other than dealing with minor SQL dialect differences that may
> exist (like any popular RDBMS dialect minor differences), the
> adaptation to Trafodion should be minimal (depending on how the
> code is structured – understand if the SQL statement are well
> separated vs have to hunt for them in many places in the code).
>
> Having said that, the statement is valid for just replacing RDBMS
> calls to trafodion. Then replacing Cassandra calls to trafodion
> should also be easy. However, if we want to replace some of
> current CQRS commands from Async to Sync, the code change will be
> more involved, as I assume the whole thing about doing “light
> check” before performing the async command, and the un-necessary
> error handling involved at the use case level (what happen if an
> async command fail, and UI thought it went OK) should be
> optionally removed. Instead classic RDBMS transaction
> commit/rollback would be done.
>
> Once implemented, is this more, less, same to maintain and add
> functionality? Would it require a dev to map the data logical
> level (ORM) more intensively? Is there a different skill set
> needed with this than current concept and are those skills
> generally available?
>
> >>> general DB skill. Easier than current solution, as standard
> SQL is easier to find than standard SQL + Cassandra special.
>
> Is the trajectory of this technology vs noSql clear?
>
> >>> Yes. It’s purpose is to add back everything that was given up
> (ACID) from classic RDBMS to achieve horizontal scalability. Now
> you have a full classic RDBMS, but capable of handling huge
> volume, with scale out share nothing architecture.
>
> You've made a good case, what's the counter?
>
> >>> The counter argument would be maturity. MySQL, Postgres or
> MariaDB, given their wide adoption, benefit from more battle
> testing than Trafodion. Having said that, and given the recent
> commitment to core banking, EsgynDB will be battle tested for core
> Banking, while the other DB are not typically seen in the
> “classic” Core Banking sector. By the time fineract-cn will be
> feature complete, I think the maturity argument will have
> seriously faded out.
>
> What about Apache Ignite?
>
> >>> Apache Ignite is a different beast. Classified as “in memory
> DB”. I have no experience with it, but can say that what is
> needed is a persistence layer, not an in memory db to speed up
> analytical queries. Therefore I have hard time seeing a fit with
> Ignite.
>
> Is there an independent review of Trafodion?
>
> >>> Not that I know of.
>
> Is the open source version useable or does this require use of the
> EsgynDB in production? Are other providers using the tech?
>
> >>> I would not use Trafodion in production, given I would never
> go in production without support contract on the database.
> Currently only Esgyn Corporation provide a Trafodion based
> production ready product.
>
> If the project devs can come to some conclusion on this, what's
> the level of effort? How should we do think about the size of this
> effort. & do you think... is there someone available to work on
> this project, perhaps from one of those banks?
>
> >>> On my free cycles, I will study fineract-cn code and analyze
> effort / prototype something.
>
> Best regards,
>
> Eric Owhadi
>
> *From:* James Dailey <jamespdailey@gmail.com
> <ma...@gmail.com>>
> *Sent:* Thursday, May 30, 2019 4:26 PM
> *To:* Dev <dev@fineract.apache.org <ma...@fineract.apache.org>>
> *Cc:* Eric Owhadi <eric.owhadi@esgyn.com
> <ma...@esgyn.com>>
> *Subject:* Re: Would it be interesting to use Apache "Trafodion"
> as an option for persistence layer?
>
> External
>
> Eric,
>
> First, thank you very much.
>
> This is a really interesting concept and proposal for a shift in
> architecture. I have some questions:
>
> What would be the impact on current efforts to remove current
> dependencies on non Apache license compatible data persistent
> layers/libraries?
>
> Once implemented, is this more, less, same to maintain and add
> functionality? Would it require a dev to map the data logical
> level (ORM) more intensively? Is there a different skill set
> needed with this than current concept and are those skills
> generally available?
>
> Is the trajectory of this technology vs noSql clear? What about
> Apache Ignite? Is there an independent review of Trafodion? You've
> made a good case, what's the counter?
>
> Is the open source version useable or does this require use of the
> EsgynDB in production? Are other providers using the tech?
>
> If the project devs can come to some conclusion on this, what's
> the level of effort? How should we do think about the size of this
> effort. & do you think... is there someone available to work on
> this project, perhaps from one of those banks?
>
> Thanks,
>
> Jdailey67
>
> On Thu, May 30, 2019, 1:48 AM Eric Owhadi
> <eric.owhadi@sbcglobal.net <ma...@sbcglobal.net>> wrote:
>
> Hello Fineracteers,
>
> I discovered Fineract and specially Fineract-cn last week, and
> am very impressed by the future potential of the project, both
> to deliver on its original social mission but also to
> revolutionize commercial FinTech sector the same way the big
> data wave caught up when Hadoop was open sourced.
>
> The architecture [Cloud Native/micro-services] is at the
> forefront of what a modern application should look like, and
> the decision to start from scratch and don’t carry the burden
> of legacy was a bold but very smart move imho.
>
> There is however, one area that might be interesting to
> challenge: the CQRS and the dual storage engine needed to
> provide capability for async command execution (and its draw
> back on error handling from front end) and sync query
> accessing the traditional DBMS with its scalability shortfall.
>
> The current implementation looks optimal when it was picked (I
> assume when fineract-cn was still mifos io), because there was
> no open source “NewSQL” solution available. Using classic
> RDBMS with known scalability limitation, coupled with NoSQL
> eventual consistency, no ACID, no multicolumn/table
> transaction, no join or grouping, given up to gain stellar
> horizontal scalability was a smart move.
>
> However, in 2017, Apache released Trafodion as Top Level
> Project. It could replace both Postgress/MariaDb/MySQL and
> Cassandra. It is an MPP RDBMS built on top of Hadoop Hbase
> (the Hadoop equivalent of Cassandra). It is fully ACID,
> supports multicolumn/multi table transactions, joins, group
> by, is fully horizontally scalable and support all the bells
> and whistle you would expect from a classic RDBMS. It can be
> classified as NewSQL, basically claiming back all the things
> given up by NoSql in exchange for scalability. In addition,
> the company backing it up (Esgyn Corp) have recently announced
> that its commercial product (EsgynDB, derived from Trafodion)
> is being used for Core Banking in 2 major Banks in China,
> struggling with the ever growing mobile payment volume that
> traditional scale up databases cannot keep up with. In
> addition Esgyn provides a DB as a service offering, cloud
> hosted fully managed (Esgyn Strato), giving the option to
> potential fineract-cn hosting company to “outsource” the
> persistence layer management and focus on the rest.
>
> Wouldn’t it be an interesting idea to hook Trafodion to
> Fineract-cn? I can think of the following advantages:
>
> * Simplify persistence layer ecosystem from 2 DB to 1 ->
> back to classic RDBMS but with full scale out capability
> MPP style.
> * Still Apache license compatible
> * Persistence layer ready to support challenging analytical
> workload (reporting use cases? - thanks for reclaiming
> Join/group-by at scale).
> * Remove the reticence of decision maker to use NoSQL in
> Core Banking
> * Performance over Cassandra [on paper, even if Trafodion
> was used only as a drop in replacement to Cassandra for
> async commands, the use of Trafodion “Align Format”
> feature should demonstrate higher performance for
> journaling, as Trafodion would write 1 BigTable cell per
> row, while Cassandra or Native Hbase would write 1 X nb
> Column cells per row].
> * With a solid company already showing its commitment to
> core banking backing the Apache project
>
> Hope this makes sense?
>
> Eric Owhadi,
>
> https://www.linkedin.com/in/eric-owhadi-3173058/
>
> Principal Engineer at Esgyn Corporation.
>
> message resent from personal email, the first tentative did
> not make it...
>
RE: Would it be interesting to use Apache "Trafodion" as an option
for persistence layer?
Posted by Eric Owhadi <er...@esgyn.com>.
Hi Graham,
Thanks you for your comment and question.
Trafodion does have a light version that we use for dev. We call it local_hadoop, and it fits in single dev boxes. This will perfectly fit this use cases,
Hope this helps,
Eric
From: Ebenezer Graham <eg...@alustudent.com>
Sent: Tuesday, June 4, 2019 7:53 AM
To: Dev <de...@fineract.apache.org>
Subject: Re: Would it be interesting to use Apache "Trafodion" as an option for persistence layer?
External
Hi Eric,
Thanks for this interesting and insightful proposal.
Please expound on how to use Trafodion in component tests and CI/CD.
If you noticed, all the services have a component-test package aside from the unit tests. This is used to test the service during development as well as for CI/CD - the data from the component tests is persisted to an embedded Cassandra and embedded MariaDB/PostgreSQL helping to mimic a similar CQRS pattern as will be used in production.
How will this work when using Trafodion? Does Trafodion have a light-weight version for this use case? If not, what is a typical approach for this?
I have looked at the specs for a basic Trafodion setup and its quite resource intensive.
Looking forward to your response
At your service,
Ebenezer Graham
On Fri, 31 May 2019 at 02:45, Eric Owhadi <er...@esgyn.com>> wrote:
Hi James,
Thanks for taking the time to answer,
Please find bellow inline answers to your questions.
What would be the impact on current efforts to remove current dependencies on non Apache license compatible data persistent layers/libraries?
>>> I am not yet familiar with the code, so can’t comment on this. What I can say is that Trafodion uses a standard jdbc driver, and other than dealing with minor SQL dialect differences that may exist (like any popular RDBMS dialect minor differences), the adaptation to Trafodion should be minimal (depending on how the code is structured – understand if the SQL statement are well separated vs have to hunt for them in many places in the code).
Having said that, the statement is valid for just replacing RDBMS calls to trafodion. Then replacing Cassandra calls to trafodion should also be easy. However, if we want to replace some of current CQRS commands from Async to Sync, the code change will be more involved, as I assume the whole thing about doing “light check” before performing the async command, and the un-necessary error handling involved at the use case level (what happen if an async command fail, and UI thought it went OK) should be optionally removed. Instead classic RDBMS transaction commit/rollback would be done.
Once implemented, is this more, less, same to maintain and add functionality? Would it require a dev to map the data logical level (ORM) more intensively? Is there a different skill set needed with this than current concept and are those skills generally available?
>>> general DB skill. Easier than current solution, as standard SQL is easier to find than standard SQL + Cassandra special.
Is the trajectory of this technology vs noSql clear?
>>> Yes. It’s purpose is to add back everything that was given up (ACID) from classic RDBMS to achieve horizontal scalability. Now you have a full classic RDBMS, but capable of handling huge volume, with scale out share nothing architecture.
You've made a good case, what's the counter?
>>> The counter argument would be maturity. MySQL, Postgres or MariaDB, given their wide adoption, benefit from more battle testing than Trafodion. Having said that, and given the recent commitment to core banking, EsgynDB will be battle tested for core Banking, while the other DB are not typically seen in the “classic” Core Banking sector. By the time fineract-cn will be feature complete, I think the maturity argument will have seriously faded out.
What about Apache Ignite?
>>> Apache Ignite is a different beast. Classified as “in memory DB”. I have no experience with it, but can say that what is needed is a persistence layer, not an in memory db to speed up analytical queries. Therefore I have hard time seeing a fit with Ignite.
Is there an independent review of Trafodion?
>>> Not that I know of.
Is the open source version useable or does this require use of the EsgynDB in production? Are other providers using the tech?
>>> I would not use Trafodion in production, given I would never go in production without support contract on the database. Currently only Esgyn Corporation provide a Trafodion based production ready product.
If the project devs can come to some conclusion on this, what's the level of effort? How should we do think about the size of this effort. & do you think... is there someone available to work on this project, perhaps from one of those banks?
>>> On my free cycles, I will study fineract-cn code and analyze effort / prototype something.
Best regards,
Eric Owhadi
From: James Dailey <ja...@gmail.com>>
Sent: Thursday, May 30, 2019 4:26 PM
To: Dev <de...@fineract.apache.org>>
Cc: Eric Owhadi <er...@esgyn.com>>
Subject: Re: Would it be interesting to use Apache "Trafodion" as an option for persistence layer?
External
Eric,
First, thank you very much.
This is a really interesting concept and proposal for a shift in architecture. I have some questions:
What would be the impact on current efforts to remove current dependencies on non Apache license compatible data persistent layers/libraries?
Once implemented, is this more, less, same to maintain and add functionality? Would it require a dev to map the data logical level (ORM) more intensively? Is there a different skill set needed with this than current concept and are those skills generally available?
Is the trajectory of this technology vs noSql clear? What about Apache Ignite? Is there an independent review of Trafodion? You've made a good case, what's the counter?
Is the open source version useable or does this require use of the EsgynDB in production? Are other providers using the tech?
If the project devs can come to some conclusion on this, what's the level of effort? How should we do think about the size of this effort. & do you think... is there someone available to work on this project, perhaps from one of those banks?
Thanks,
Jdailey67
On Thu, May 30, 2019, 1:48 AM Eric Owhadi <er...@sbcglobal.net>> wrote:
Hello Fineracteers,
I discovered Fineract and specially Fineract-cn last week, and am very impressed by the future potential of the project, both to deliver on its original social mission but also to revolutionize commercial FinTech sector the same way the big data wave caught up when Hadoop was open sourced.
The architecture [Cloud Native/micro-services] is at the forefront of what a modern application should look like, and the decision to start from scratch and don’t carry the burden of legacy was a bold but very smart move imho.
There is however, one area that might be interesting to challenge: the CQRS and the dual storage engine needed to provide capability for async command execution (and its draw back on error handling from front end) and sync query accessing the traditional DBMS with its scalability shortfall.
The current implementation looks optimal when it was picked (I assume when fineract-cn was still mifos io), because there was no open source “NewSQL” solution available. Using classic RDBMS with known scalability limitation, coupled with NoSQL eventual consistency, no ACID, no multicolumn/table transaction, no join or grouping, given up to gain stellar horizontal scalability was a smart move.
However, in 2017, Apache released Trafodion as Top Level Project. It could replace both Postgress/MariaDb/MySQL and Cassandra. It is an MPP RDBMS built on top of Hadoop Hbase (the Hadoop equivalent of Cassandra). It is fully ACID, supports multicolumn/multi table transactions, joins, group by, is fully horizontally scalable and support all the bells and whistle you would expect from a classic RDBMS. It can be classified as NewSQL, basically claiming back all the things given up by NoSql in exchange for scalability. In addition, the company backing it up (Esgyn Corp) have recently announced that its commercial product (EsgynDB, derived from Trafodion) is being used for Core Banking in 2 major Banks in China, struggling with the ever growing mobile payment volume that traditional scale up databases cannot keep up with. In addition Esgyn provides a DB as a service offering, cloud hosted fully managed (Esgyn Strato), giving the option to potential fineract-cn hosting company to “outsource” the persistence layer management and focus on the rest.
Wouldn’t it be an interesting idea to hook Trafodion to Fineract-cn? I can think of the following advantages:
* Simplify persistence layer ecosystem from 2 DB to 1 -> back to classic RDBMS but with full scale out capability MPP style.
* Still Apache license compatible
* Persistence layer ready to support challenging analytical workload (reporting use cases? - thanks for reclaiming Join/group-by at scale).
* Remove the reticence of decision maker to use NoSQL in Core Banking
* Performance over Cassandra [on paper, even if Trafodion was used only as a drop in replacement to Cassandra for async commands, the use of Trafodion “Align Format” feature should demonstrate higher performance for journaling, as Trafodion would write 1 BigTable cell per row, while Cassandra or Native Hbase would write 1 X nb Column cells per row].
* With a solid company already showing its commitment to core banking backing the Apache project
Hope this makes sense?
Eric Owhadi,
https://www.linkedin.com/in/eric-owhadi-3173058/
Principal Engineer at Esgyn Corporation.
message resent from personal email, the first tentative did not make it...