You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@fineract.apache.org by Isaac Kamga <is...@mifos.org> on 2019/01/26 23:45:53 UTC
One-process deployment of Fineract CN
Happy weekend fineracters,
Hope this email finds you in good health.
In an effort to facilitate local development of Fineract CN, I picked up
the One-process deployment issue [1] which when resolved can enable
developers to run the platform in a single process.
Some research led me to the possibility of running each microservice in a
it's own ClassLoader while ensuring that the ServiceRunner uses a single
JVM to run in a single process. After having tried to adapt the method in
[2] where several Spring Boot apps are launched in the same process to our
Service-starter and demo-server, I encountered this error [3]. The code of
the workdone so far is on the *oneprocess* branch of my fork of
service-starter[4] and demo-server [5].
@Myrle Krantz <my...@apache.org> Is there a way to have the "java -jar
xyz-service-boot-0.1.0-BUILD-SNAPSHOT.jar" command run a single Thread as
opposed to running in the process assigned by ProcessBuilder() ?
@Markus Geiss <ma...@apache.org> , is it possible that what could help in
this case is using an entire Pool of threads to get this executed ? Asking
as you developed the fineract-cn-async [6] library and I observed that upon
running demo-server, when the processors on my machine get exhausted, the
logs indicate that a separate thread (not main) shows up to run any
available service like Notifications and Groups.
Any help from the community would be much appreciated.
At Your Service,
Isaac Kamga.
[1] https://issues.apache.org/jira/browse/FINCN-25
[2]
https://github.com/rameez4ever/springboot-demo/tree/master/springboot-multi-service-launcher
[3] https://pastebin.com/qBDW52Y5
[4] https://github.com/Izakey/fineract-cn-service-starter/tree/oneprocess
[5] https://github.com/Izakey/fineract-cn-demo-server/tree/oneprocess
[6] https://github.com/apache/fineract-cn-async
Re: One-process deployment of Fineract CN
Posted by Isaac Kamga <is...@mifos.org>.
Hello Myrle, Vishwas,
@Myrle, Thanks very much for elucidating on your approach.
@Vishwas, Good work ! I've tested your PR with *-Ddemoserver.lite=true*
option and tail of the logs showed
"*identity-v1 address:http://localhost:2021/identity/v1
<http://localhost:2021/identity/v1>*
*office-v1 address:http://localhost:2023/office/v1
<http://localhost:2023/office/v1>*
*customer-v1 address:http://localhost:2024/customer/v1
<http://localhost:2024/customer/v1>*" and it's merged.
At Your Service,
Isaac Kamga.
On Mon, Feb 11, 2019 at 2:32 AM Vishwas Babu <
vishwas@confluxtechnologies.com> wrote:
> Hi Myrle, Issac,
>
> For folks like me who are trying to have a quick look at the architecture
> and provisioning process in Fineract-CN, having an option for bringing up a
> few micro-services relatively quickly using the demo server would be
> helpful. The pr at
> https://github.com/apache/fineract-cn-demo-server/pull/25
> adds the ability to restrict the working set of microservices
> to Provisioner, Identity, Rhythm, Organization and Customer.
>
> @James Dailey <JA...@gmail.com> : Maybe this might work for you
> too? Running the demo server for the first time with demoserver.lite flag
> turned on takes less than 9 minutes for me on an iMac having a local
> Cassandra, MySQL and ActiveMQ and sporting a 2.5GHz Core i5 (late 2014)
> processor and 24 GB of RAM.
>
> Regards,
> Vishwas
>
>
>
> On Fri, Feb 8, 2019 at 11:18 PM Myrle Krantz <my...@apache.org> wrote:
>
> > My idea was to produce jars without mains and then run them all from one
> > main. This would mean that they all run under the same service name.
> This
> > in turn makes it necessary to solve problems around collisions:
> >
> > * In most cases endpoints don't have the same names, but there's an
> > "iniitialize" in every service.
> > * permissions would need to include the name of the service in
> > the @Permittable annotation.
> >
> > It sounds like a small set of changes, but it's actually quite extensive.
> > The changes to @Permittable require a lot of re-engineering in anubis.
> >
> > If one process can run multiple services and keep their names, that's so
> > much better.
> >
> > Best Regards,
> > Myrle
> >
> > On Fri, Feb 8, 2019 at 3:39 AM Isaac Kamga <is...@mifos.org>
> wrote:
> >
> > >
> > > Myrle, thanks a million for your input. While I'm digging into ways to
> > fix
> > > the current errors, can you kindly share the other approach which you
> > > thought would be less promising to solve this issue ? Maybe others
> > > interested could try that out in parallel too.
> > >
> > > Cheers,
> > > Isaac Kamga.
> > >
> >
>
Re: One-process deployment of Fineract CN
Posted by Vishwas Babu <vi...@confluxtechnologies.com>.
Hi Myrle, Issac,
For folks like me who are trying to have a quick look at the architecture
and provisioning process in Fineract-CN, having an option for bringing up a
few micro-services relatively quickly using the demo server would be
helpful. The pr at https://github.com/apache/fineract-cn-demo-server/pull/25
adds the ability to restrict the working set of microservices
to Provisioner, Identity, Rhythm, Organization and Customer.
@James Dailey <JA...@gmail.com> : Maybe this might work for you
too? Running the demo server for the first time with demoserver.lite flag
turned on takes less than 9 minutes for me on an iMac having a local
Cassandra, MySQL and ActiveMQ and sporting a 2.5GHz Core i5 (late 2014)
processor and 24 GB of RAM.
Regards,
Vishwas
On Fri, Feb 8, 2019 at 11:18 PM Myrle Krantz <my...@apache.org> wrote:
> My idea was to produce jars without mains and then run them all from one
> main. This would mean that they all run under the same service name. This
> in turn makes it necessary to solve problems around collisions:
>
> * In most cases endpoints don't have the same names, but there's an
> "iniitialize" in every service.
> * permissions would need to include the name of the service in
> the @Permittable annotation.
>
> It sounds like a small set of changes, but it's actually quite extensive.
> The changes to @Permittable require a lot of re-engineering in anubis.
>
> If one process can run multiple services and keep their names, that's so
> much better.
>
> Best Regards,
> Myrle
>
> On Fri, Feb 8, 2019 at 3:39 AM Isaac Kamga <is...@mifos.org> wrote:
>
> >
> > Myrle, thanks a million for your input. While I'm digging into ways to
> fix
> > the current errors, can you kindly share the other approach which you
> > thought would be less promising to solve this issue ? Maybe others
> > interested could try that out in parallel too.
> >
> > Cheers,
> > Isaac Kamga.
> >
>
Re: One-process deployment of Fineract CN
Posted by Myrle Krantz <my...@apache.org>.
My idea was to produce jars without mains and then run them all from one
main. This would mean that they all run under the same service name. This
in turn makes it necessary to solve problems around collisions:
* In most cases endpoints don't have the same names, but there's an
"iniitialize" in every service.
* permissions would need to include the name of the service in
the @Permittable annotation.
It sounds like a small set of changes, but it's actually quite extensive.
The changes to @Permittable require a lot of re-engineering in anubis.
If one process can run multiple services and keep their names, that's so
much better.
Best Regards,
Myrle
On Fri, Feb 8, 2019 at 3:39 AM Isaac Kamga <is...@mifos.org> wrote:
>
> Myrle, thanks a million for your input. While I'm digging into ways to fix
> the current errors, can you kindly share the other approach which you
> thought would be less promising to solve this issue ? Maybe others
> interested could try that out in parallel too.
>
> Cheers,
> Isaac Kamga.
>
Re: One-process deployment of Fineract CN
Posted by Isaac Kamga <is...@mifos.org>.
Hey Devs,
Kelvin, thanks for your reply. As you're well aware, just a handful of
fineracters have had the chance to enjoy running the services in
production.
Myrle, thanks a million for your input. While I'm digging into ways to fix
the current errors, can you kindly share the other approach which you
thought would be less promising to solve this issue ? Maybe others
interested could try that out in parallel too.
Cheers,
Isaac Kamga.
On Mon, Feb 4, 2019 at 10:41 AM Myrle Krantz <my...@apache.org> wrote:
> Kelvin,
>
> The problem Isaac is trying to solve is that many of our contributors don’t
> have enough memory or fast enough swap speeds to run the services all at
> once on their computers for doing development testing. Also startup speeds
> are very slow because some services need to be completely up and
> provisioned before other services can be started.
>
> Isaac is not trying to create a production environment here. If he were,
> than the rate at which requests can be handled would, of course, matter.
> But demo-server was never intended as a production environment; correct
> production environment will be *highly* dependent on use case.
>
> Nonetheless I would welcome your contribution of docker specifications. I
> do think the docker files should go into the service repositories rather
> than the demo-server. There are also exciting ideas in circulation for
> doing this with OpenShift instead.
>
> Best Regards,
> Myrle
>
> On Sat, Feb 2, 2019 at 9:04 PM Kelvin Ikome <kv...@gmail.com> wrote:
>
> > Hi Isaac,
> >
> > Technically, running the services within a single process should reduce
> > required number of virtual processors (cores). However, this might also
> > lead to a rise in processing speed.
> >
> > For example, if using one process per service requires about 2.0Ghz Quad
> > core processing power, running the very same services using a single
> thread
> > per service might require about 3.0Ghz Dual core processing power.
> >
> > Using multi threading to run the microservices within a single process
> is a
> > great idea and will help simplify the process of running the platform.
> > However, I don't think this might be the best approach to optimize the
> > required processing power. The most effective way to run the platform I
> can
> > think of would be to ditch demo server and run the services within docker
> > containers. In addition to that, using containers gives you the
> flexibility
> > of working with each individual service in a sort of "plug and play"
> manner
> > as well.
> >
> > Just my thoughts :)
> >
> > On Mon, Jan 28, 2019 at 8:54 PM Myrle Krantz <my...@apache.org> wrote:
> >
> > > Sorry Isaac,
> > >
> > > can't help you here. I had an entirely different idea in mind for how
> to
> > > do this, and yours looks more promising, actually. But I can't tell
> you
> > > why it's not working.
> > >
> > > Best Regards,
> > > Myrle
> > >
> > > On Sun, Jan 27, 2019 at 12:46 AM Isaac Kamga <is...@mifos.org>
> > > wrote:
> > >
> > > > Happy weekend fineracters,
> > > >
> > > > Hope this email finds you in good health.
> > > >
> > > > In an effort to facilitate local development of Fineract CN, I picked
> > up
> > > > the One-process deployment issue [1] which when resolved can enable
> > > > developers to run the platform in a single process.
> > > >
> > > > Some research led me to the possibility of running each microservice
> > in a
> > > > it's own ClassLoader while ensuring that the ServiceRunner uses a
> > single
> > > > JVM to run in a single process. After having tried to adapt the
> method
> > in
> > > > [2] where several Spring Boot apps are launched in the same process
> to
> > > our
> > > > Service-starter and demo-server, I encountered this error [3]. The
> code
> > > of
> > > > the workdone so far is on the *oneprocess* branch of my fork of
> > > > service-starter[4] and demo-server [5].
> > > >
> > > > @Myrle Krantz <my...@apache.org> Is there a way to have the "java
> > -jar
> > > > xyz-service-boot-0.1.0-BUILD-SNAPSHOT.jar" command run a single
> Thread
> > as
> > > > opposed to running in the process assigned by ProcessBuilder() ?
> > > >
> > > > @Markus Geiss <ma...@apache.org> , is it possible that what could
> help
> > in
> > > > this case is using an entire Pool of threads to get this executed ?
> > > Asking
> > > > as you developed the fineract-cn-async [6] library and I observed
> that
> > > upon
> > > > running demo-server, when the processors on my machine get exhausted,
> > the
> > > > logs indicate that a separate thread (not main) shows up to run any
> > > > available service like Notifications and Groups.
> > > >
> > > > Any help from the community would be much appreciated.
> > > >
> > > > At Your Service,
> > > > Isaac Kamga.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FINCN-25
> > > > [2]
> > > >
> > >
> >
> https://github.com/rameez4ever/springboot-demo/tree/master/springboot-multi-service-launcher
> > > >
> > > > [3] https://pastebin.com/qBDW52Y5
> > > > [4]
> > > https://github.com/Izakey/fineract-cn-service-starter/tree/oneprocess
> > > > [5]
> https://github.com/Izakey/fineract-cn-demo-server/tree/oneprocess
> > > > [6] https://github.com/apache/fineract-cn-async
> > > >
> > >
> >
>
Re: One-process deployment of Fineract CN
Posted by Myrle Krantz <my...@apache.org>.
Kelvin,
The problem Isaac is trying to solve is that many of our contributors don’t
have enough memory or fast enough swap speeds to run the services all at
once on their computers for doing development testing. Also startup speeds
are very slow because some services need to be completely up and
provisioned before other services can be started.
Isaac is not trying to create a production environment here. If he were,
than the rate at which requests can be handled would, of course, matter.
But demo-server was never intended as a production environment; correct
production environment will be *highly* dependent on use case.
Nonetheless I would welcome your contribution of docker specifications. I
do think the docker files should go into the service repositories rather
than the demo-server. There are also exciting ideas in circulation for
doing this with OpenShift instead.
Best Regards,
Myrle
On Sat, Feb 2, 2019 at 9:04 PM Kelvin Ikome <kv...@gmail.com> wrote:
> Hi Isaac,
>
> Technically, running the services within a single process should reduce
> required number of virtual processors (cores). However, this might also
> lead to a rise in processing speed.
>
> For example, if using one process per service requires about 2.0Ghz Quad
> core processing power, running the very same services using a single thread
> per service might require about 3.0Ghz Dual core processing power.
>
> Using multi threading to run the microservices within a single process is a
> great idea and will help simplify the process of running the platform.
> However, I don't think this might be the best approach to optimize the
> required processing power. The most effective way to run the platform I can
> think of would be to ditch demo server and run the services within docker
> containers. In addition to that, using containers gives you the flexibility
> of working with each individual service in a sort of "plug and play" manner
> as well.
>
> Just my thoughts :)
>
> On Mon, Jan 28, 2019 at 8:54 PM Myrle Krantz <my...@apache.org> wrote:
>
> > Sorry Isaac,
> >
> > can't help you here. I had an entirely different idea in mind for how to
> > do this, and yours looks more promising, actually. But I can't tell you
> > why it's not working.
> >
> > Best Regards,
> > Myrle
> >
> > On Sun, Jan 27, 2019 at 12:46 AM Isaac Kamga <is...@mifos.org>
> > wrote:
> >
> > > Happy weekend fineracters,
> > >
> > > Hope this email finds you in good health.
> > >
> > > In an effort to facilitate local development of Fineract CN, I picked
> up
> > > the One-process deployment issue [1] which when resolved can enable
> > > developers to run the platform in a single process.
> > >
> > > Some research led me to the possibility of running each microservice
> in a
> > > it's own ClassLoader while ensuring that the ServiceRunner uses a
> single
> > > JVM to run in a single process. After having tried to adapt the method
> in
> > > [2] where several Spring Boot apps are launched in the same process to
> > our
> > > Service-starter and demo-server, I encountered this error [3]. The code
> > of
> > > the workdone so far is on the *oneprocess* branch of my fork of
> > > service-starter[4] and demo-server [5].
> > >
> > > @Myrle Krantz <my...@apache.org> Is there a way to have the "java
> -jar
> > > xyz-service-boot-0.1.0-BUILD-SNAPSHOT.jar" command run a single Thread
> as
> > > opposed to running in the process assigned by ProcessBuilder() ?
> > >
> > > @Markus Geiss <ma...@apache.org> , is it possible that what could help
> in
> > > this case is using an entire Pool of threads to get this executed ?
> > Asking
> > > as you developed the fineract-cn-async [6] library and I observed that
> > upon
> > > running demo-server, when the processors on my machine get exhausted,
> the
> > > logs indicate that a separate thread (not main) shows up to run any
> > > available service like Notifications and Groups.
> > >
> > > Any help from the community would be much appreciated.
> > >
> > > At Your Service,
> > > Isaac Kamga.
> > >
> > > [1] https://issues.apache.org/jira/browse/FINCN-25
> > > [2]
> > >
> >
> https://github.com/rameez4ever/springboot-demo/tree/master/springboot-multi-service-launcher
> > >
> > > [3] https://pastebin.com/qBDW52Y5
> > > [4]
> > https://github.com/Izakey/fineract-cn-service-starter/tree/oneprocess
> > > [5] https://github.com/Izakey/fineract-cn-demo-server/tree/oneprocess
> > > [6] https://github.com/apache/fineract-cn-async
> > >
> >
>
Re: One-process deployment of Fineract CN
Posted by Kelvin Ikome <kv...@gmail.com>.
Hi Isaac,
Technically, running the services within a single process should reduce
required number of virtual processors (cores). However, this might also
lead to a rise in processing speed.
For example, if using one process per service requires about 2.0Ghz Quad
core processing power, running the very same services using a single thread
per service might require about 3.0Ghz Dual core processing power.
Using multi threading to run the microservices within a single process is a
great idea and will help simplify the process of running the platform.
However, I don't think this might be the best approach to optimize the
required processing power. The most effective way to run the platform I can
think of would be to ditch demo server and run the services within docker
containers. In addition to that, using containers gives you the flexibility
of working with each individual service in a sort of "plug and play" manner
as well.
Just my thoughts :)
On Mon, Jan 28, 2019 at 8:54 PM Myrle Krantz <my...@apache.org> wrote:
> Sorry Isaac,
>
> can't help you here. I had an entirely different idea in mind for how to
> do this, and yours looks more promising, actually. But I can't tell you
> why it's not working.
>
> Best Regards,
> Myrle
>
> On Sun, Jan 27, 2019 at 12:46 AM Isaac Kamga <is...@mifos.org>
> wrote:
>
> > Happy weekend fineracters,
> >
> > Hope this email finds you in good health.
> >
> > In an effort to facilitate local development of Fineract CN, I picked up
> > the One-process deployment issue [1] which when resolved can enable
> > developers to run the platform in a single process.
> >
> > Some research led me to the possibility of running each microservice in a
> > it's own ClassLoader while ensuring that the ServiceRunner uses a single
> > JVM to run in a single process. After having tried to adapt the method in
> > [2] where several Spring Boot apps are launched in the same process to
> our
> > Service-starter and demo-server, I encountered this error [3]. The code
> of
> > the workdone so far is on the *oneprocess* branch of my fork of
> > service-starter[4] and demo-server [5].
> >
> > @Myrle Krantz <my...@apache.org> Is there a way to have the "java -jar
> > xyz-service-boot-0.1.0-BUILD-SNAPSHOT.jar" command run a single Thread as
> > opposed to running in the process assigned by ProcessBuilder() ?
> >
> > @Markus Geiss <ma...@apache.org> , is it possible that what could help in
> > this case is using an entire Pool of threads to get this executed ?
> Asking
> > as you developed the fineract-cn-async [6] library and I observed that
> upon
> > running demo-server, when the processors on my machine get exhausted, the
> > logs indicate that a separate thread (not main) shows up to run any
> > available service like Notifications and Groups.
> >
> > Any help from the community would be much appreciated.
> >
> > At Your Service,
> > Isaac Kamga.
> >
> > [1] https://issues.apache.org/jira/browse/FINCN-25
> > [2]
> >
> https://github.com/rameez4ever/springboot-demo/tree/master/springboot-multi-service-launcher
> >
> > [3] https://pastebin.com/qBDW52Y5
> > [4]
> https://github.com/Izakey/fineract-cn-service-starter/tree/oneprocess
> > [5] https://github.com/Izakey/fineract-cn-demo-server/tree/oneprocess
> > [6] https://github.com/apache/fineract-cn-async
> >
>
Re: One-process deployment of Fineract CN
Posted by Myrle Krantz <my...@apache.org>.
Sorry Isaac,
can't help you here. I had an entirely different idea in mind for how to
do this, and yours looks more promising, actually. But I can't tell you
why it's not working.
Best Regards,
Myrle
On Sun, Jan 27, 2019 at 12:46 AM Isaac Kamga <is...@mifos.org> wrote:
> Happy weekend fineracters,
>
> Hope this email finds you in good health.
>
> In an effort to facilitate local development of Fineract CN, I picked up
> the One-process deployment issue [1] which when resolved can enable
> developers to run the platform in a single process.
>
> Some research led me to the possibility of running each microservice in a
> it's own ClassLoader while ensuring that the ServiceRunner uses a single
> JVM to run in a single process. After having tried to adapt the method in
> [2] where several Spring Boot apps are launched in the same process to our
> Service-starter and demo-server, I encountered this error [3]. The code of
> the workdone so far is on the *oneprocess* branch of my fork of
> service-starter[4] and demo-server [5].
>
> @Myrle Krantz <my...@apache.org> Is there a way to have the "java -jar
> xyz-service-boot-0.1.0-BUILD-SNAPSHOT.jar" command run a single Thread as
> opposed to running in the process assigned by ProcessBuilder() ?
>
> @Markus Geiss <ma...@apache.org> , is it possible that what could help in
> this case is using an entire Pool of threads to get this executed ? Asking
> as you developed the fineract-cn-async [6] library and I observed that upon
> running demo-server, when the processors on my machine get exhausted, the
> logs indicate that a separate thread (not main) shows up to run any
> available service like Notifications and Groups.
>
> Any help from the community would be much appreciated.
>
> At Your Service,
> Isaac Kamga.
>
> [1] https://issues.apache.org/jira/browse/FINCN-25
> [2]
> https://github.com/rameez4ever/springboot-demo/tree/master/springboot-multi-service-launcher
>
> [3] https://pastebin.com/qBDW52Y5
> [4] https://github.com/Izakey/fineract-cn-service-starter/tree/oneprocess
> [5] https://github.com/Izakey/fineract-cn-demo-server/tree/oneprocess
> [6] https://github.com/apache/fineract-cn-async
>