You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ofbiz.apache.org by Taher Alkhateeb <ta...@pythys.com.INVALID> on 2021/12/22 17:23:28 UTC
OFBiz re-architecture thoughts
Hello Folks,
I love OFBiz. I met great people and I learned a lot. Our business would
not have existed nor succeeded without being introduced to OFBiz and the
community.
In the past couple of years, we hesitantly started switching projects
out of OFBiz due to multiple issues. I think these issues cannot be
easily solved without some kind of a major re-architecture of the
framework and so the purpose of this thread is to start a discussion to
see if there is interest / ideas for moving in that direction (so it's a
Hail Mary Pass).
I'm going to list in here:
- Problems
- Solution ideas
- Keep It
- Improve It
- Add It
- Remove It
- Quality control
- Tradeoffs
- Pros
- Cons
- Summary
Problems
I've identified problems not only from our past work with OFBiz but also
by working extensively with moqui (as a point of comparison) in the past
few years. Listing them below:*
*
_1. Lack of Code Isolation_:It is unlikely to do anything substantial in
OFBiz without touching the core repositories. Compare this with moqui or
some other web frameworks like say Ruby-On-Rails or django and you will
see that you can work independently of the development path in these
frameworks. This makes upgrades quite painfulwith patches and manual
interventions. Some in the community even introduced a whole patching
framework in the past.
_2. All or nothing:_In OFBiz, if you want to use the system, either take
the whole thing or take nothing. What if I don't want any user interface
for example, and just want to utilize the entities and services?
_3. Limited Re-usability:_Because of the way current components are
designed, it's difficult to take advantage of components in other
components. Sharing artifacts is pretty much limited to entities and
services. Even then they're not very shareable because they're designed
in a non-usable way for a certain context and certain settings and
database parameters.*
*
_4. Fragmented Actions:_**Instead of having unified XML actions, you
have fragmentation across different contexts (screens, services, SECAs,
etc). Furthermore, you cannot mix XML actions with groovy or other
artifacts except by calling into external scripts and things like that.
In comparison to moqui which is more flexible in XML actions I find it
restrictive to work with OFBiz actions and it's hard to review, debug,
or modify. I'm listing something below as an example for comparison [1]*
*
_5. Limited Routing capability:_**Routing is very minimal
(controller.xml). It cannot contain business logic and is limited either
to chaining requests or rendering screens. Sophisticated routing logic
is important for making rich systems that react to differing circumstances.
**
_6. Core framework difficult to change:_**OFBiz core was written
a/long/time ago, and back then you didn't have many libraries for
caching, transactions, DB pooling, Concurrency stuff, and others, so a
lot was hand crafted, and we have a lots of interconnected API that is
difficult to change, and most of these APIs depend on static util files
that just kept piling and piling over the years. Where is the service
engine implementation exactly? What about the entity engine? What about
widgets? They are all over the place! Pieces for XML parsing, others for
DOM models, and so on. The code is scattered and fixing it means going
through a maze of many classes that are interconnected and all
referencing each other. It's also hard to identify the signature or
figure out the "big picture" since you have essentially minimal
interfaces and no clear public API that you can refactor against. Pretty
much the whole thing is public API**[2].
**
_7. Web-apps design problem__:_ A big problem (in my opinion) in the
design is around having multiple web-apps, one for each component (or
more). This interrupts the flow of the app and makes it difficult to
come up with smooth solutions that work across the various screens and
instead we need to pass a key between the components (and face other
problems around session management). This restricts you in what you can
offer as a user experience.*
*
_8. Problematic UI design__:_ Another/big/problem (in my opinion) is
having the UI show everything (reference implementation). I never
understood the value of showing/everything/! If I want to understand the
data model, I will just look at the data model. The purpose of the UI is
not to show everything, but to implement specific useful use-cases. The
reference implementation we have is not doing that, and not offering
something very useful, instead it is showing everything! To improve
this, we need to switch from data-model based interface (party, order,
etc ...) to use-case based interface (project management, e-commerce,
ERP, CRM, etc ...)*
*
_9. Limited client rendering:_Because it is difficult to refactor the
framework, it's hard to pin-point where to go exactly to refactor all
the code needed to support more client-rendering technology (react, vue,
angular, etc ...) and make it interact smoothly with the back-end
(drop-downs, complex forms, etc ...). A lot of work needs to happen both
to the /themes and to /framework to allow and provide more flexibility
(also moving things to DB). If this step is not done you will be stuck
with your UI capabilities and won't catch up (unless you develop your
own UI which makes the UI of the community redundant and you let go of
the whole widget system anyway).*
*
_10. Security is difficult to implement:_One of my favorite features in
moqui is having security baked in the artifacts themselves to avoid
sprinkling all screens / services with security logic. I never even
worry about writing security related stuff until way close to the end of
delivering the project. So there needs to be a whole security layer that
sits below all the artifacts such that we don't even need to write
security code in the various screens and services and around certain
entities.*
*
_11. Limited Integration Solutions__:_ I know we have now a REST
component, but this is not the same as being a core feature of the
framework (essential for any modern web framework). Why should it be
tightly integrated? Many reasons: security, localization, mapping to
service engine, mapping to entities, etc ...*
*
_12. No Specialization:_Because of the big repository, and having
everything point to everything, it's hard to specialize. Now compare
this to the approach we have in our work which is easier to specialize
in. For example, I have a component for payment gateway integration,
another one for twilio (SMS integration), and I have application
components for each solution in its own repository (HR, insurance stuff,
many others) and each is living in its own git repo yet they work
together just fine and without touching one line of code in the
framework or even core components (despite depending on them and using
pieces of them). I can jump into new versions of the framework everyday
without breaking a sweat. When I work on component X, I'm only thinking
of that and nothing else. I don't have to worry about changing something
that would crash another component or bring the system down (happened to
me many times in the past).
Solutions
Whatever solutions we come up with I think they're going to be difficult
and would require community buy-in and support. I'm breaking this down
into things to keep, update, add and remove as follows:
Keep it
- XSD signatures (so entity definitions, service definitions, XML
actions though should expand)
- Basic constructs of what is a component (with basic directory structure)
- Data model
- Existing services (core data-model services)
- Java / groovy code that doesn't use core framework constructs or goes
too deep into the internals.
- Most used and useful APIs (Making queries, calling services, etc ...)
although it should all be converted to interfaces with separate
implementations where it's lacking.
Improve It
- Break the project to smaller git repositories.
- framework
- data model
- services
- common UI
- use-case applications (ERP, CRM, e-commerce, etc ...)
- plugins
- entire framework directory needs refurbishment and big chunks probably
require a delete / rewrite
- the web architecture (get rid of web-apps)
- the tests (lots of them need to be deleted anyway)
- Routing needs to be programmatic with ability to inject code.
Add It
- Configuration override mechanism to remove the need to touch the core.
Overrides include everything including routing, component registration,
etc ...
- REST integration into the core
- Artifact based security that maps users / groups to artifacts without
the need to inject security code into every screen
- A super-set of XML actions that can be applied in all contexts and can
embed groovy code
- New, simple use-case based user interface. It needs to be designed
such that it is easy to extend to avoid the all-or-nothing UI that
currently exists.
Remove It / Deprecate It / Overwrite It
- Most of the existing UI (needs to be replaced with simpler use-case
based interface)
- Lots of the core framework code (needs major rewrites, needs
interfaces, import libraries for things like transactions, security,
concurrency, etc ...)
- Probably some of the existing plugins might need to be deprecated.
Quality Control
Investing in such a huge effort requires that we change the way we view
the project after / while work is being completed. Code should be
approached with greater care and a lot of emphasis should be put on high
quality, readability, avoiding anti-patterns, enforcing good patterns
and best-practices. Some of the daily workflows need to improve /
change, more thorough auditing and peer-reviews, and so on.
Tradeoffs
It goes without saying that the above is a _lot_ of work and commitment
and requires a lot of effort. That's why I hesitated in starting this
thread, but I thought maybe showing the tradeoffs would make it easier
to weigh things out and decide on a direction to push.
Pros
- Pretty much most of the above mentioned problems will be resolved.
- The framework would be resilient and future proof.
- We will have more options and capabilities in the future, it would be
easier to incorporate new technologies as they emerge
- Development would go faster
- Technical Debt will be reduced
Cons
- Huge (maybe too huge) upfront costs in time and effort to make the switch
- Extra effort needed to ensure a smooth upgrade path for existing users
/ adopters / system integrators. A whole layer of code needs to be
written to ensure the ability to upgrade without massive rewrites.
- Risk of failure, exhaustion, inability to push through to the end
- Inconvenience to those with existing systems / solutions
Summary
Change is the only constant. Inability to adapt to the future means not
surviving. I'm not confident what is the best change possible or at what
pace, but I do know that lack of change is a death sentence to any
project. Maybe it's time to reflect and think of where we'd like to go
and this thread is sort of a brain-dump of possible ideas.
I look forward to hearing your thoughts.
Cheers,
Taher Alkhateeb
[1] Example mixed XML actions and groovy from moqui
<if condition="settlementTerm?.orderPmtServiceRegisterId">
<then>
<service-call name="update#mantle.order.OrderPart"
in-map="[orderId:orderId, orderPartSeqId:orderPartSeqId, settlementTermId:settlementTermId]"/>
<entity-find-one entity-name="moqui.service.ServiceRegister" value-field="serviceRegister">
<field-map field-name="serviceRegisterId" from="settlementTerm.orderPmtServiceRegisterId"/>
</entity-find-one>
<script><![CDATA[
try {
ec.service.sync().name(serviceRegister.serviceName).parameters(context).softValidate(true).disableAuthz().call()
if (ec.message.hasError()) return
} catch (Throwable t) {
ec.logger.log(300, "Error calling auto order payments service for SettlementTerm ${settlementTermId}", t)
}
]]></script>
</then>
<else>
<service-call name="mantle.account.PaymentServices.create#Payment" out-map="context"
in-map="context + [amount:amount, amountUomId:amountUomId, fromPartyId:fromPartyId, toPartyId:toPartyId]"/>
</else>
</if>
[2] https://ci.apache.org/projects/ofbiz/site/trunk/javadocs/
Re: OFBiz re-architecture thoughts
Posted by Sathish Kumar Thiyagarajan <sa...@gmail.com>.
Dear Taher/Michael,
There are two schools of thoughts when it comes to upgrades.
1. Oneside, Developers who have put so much of an effort on the product
will be reluctant to change (until we see logical,visible and/or immediate
benifits).
2. Another side, Who wants to be on the cutting edge. to take advanced
features.
I have been part of some large scale migration. My experience teaches me
that none of the above are wrong and at the same time none of the above are
correct too. Might look a bit confusing but it is the reality. Before we go
to success aspects of migration, let me first put the failures i have
scenes with these approaches
1. Projects that provided services for a long time become relevant in a
short span of time due advanced and useful new competitors.
2. Projects that went over ambitious with cutting edge technology,
messed up the whole product due to poor execution.
Having said that, no software can stay away from migration or upgrades
forever. It has to happen today/tomorrow or later. What is in our hands is
to make it planned one or forced one. One common aspect that I have seen in
successful migration is that most of them are not done in a day or two. The
plan and execution happens as part of everyday work.
We shall start with small but effective steps first. May be a dedicated
page (if not available already) and collect the wishlist from developers
and users of ofbiz.
Thoughts ?!
Regards,
Sathish Kumar Thiyagarajan
On Mon, Dec 27, 2021 at 7:51 PM Michael Brohl <mi...@ecomify.de>
wrote:
> Hi Taher,
>
> thanks for bringing this up! I will try to briefly explain my views on
> the raised topics.
>
> My perspective is from nearly 20 years of intensive work with OFBiz,
> both for internal and external projects. Those projects have/had a
> volume reaching from about 50 to over 1000 project days and we have
> active customer projects which are running for almost 20 years now,
> still being developed further with the business. OFBiz often became a
> crucial part of the business ecosystem of those users. It is also the
> central toolbox for our services around development of web based
> software solutions and system integration in our own company since 7
> years now. I consider myself both a user of OFBiz and a service
> provider/integrator.
>
> After so many years of working with OFBiz if have learned that - from a
> business users perspective - the functionality and integration aspects
> along with the powerful datamodel, services etc. as well as flexibility,
> performance and scalability is way more important than detailed
> technical aspects (in the sense of “modern” or “newest technology”).
>
> As a service provider for OFBiz, we are feeling responsible to provide a
> reliable, constant and upgradeable software ecosystem which is why we
> are on a more "conservative" path regarding big or long running changes
> in the project. I have seen several new, ambititious contributors who
> wanted to change everything, using the newest frameworks and tools
> around etc., but often did not have an extensive background of real
> world projects or who are not able to explain why they want to use other
> tools and mechanisms and which user problems they really want to solve.
>
> Most of these people are not active anymore or unfinished solutions are
> put into the codebase which is another reason why I am really careful
> when it comes to massive changes. I think the project needs good
> planning, a good migration path and constant support when those changes
> are decided to take place.
>
> All in all, I do not feel such a great pain as you seem to feel but
> there is definetely much room for improvement, for sure.
>
> I will try to give my views on the different topics inline. They are
> brief and need/deserve more in-depth preparation from my side which I
> cannot spare at the moment.
>
> For further discussions I recommend to split those topics in different
> threads because they would become unreadable soon with more
> participants. Also writing up the outcome of the discussions and the
> further roadmap in the wiki might help.
>
> Let's see how it works.
>
> Michael Brohl
>
> ecomify GmbH - www.ecomify.de
>
>
> Am 22.12.21 um 18:23 schrieb Taher Alkhateeb:
> > Problems
> ...
> > _1. Lack of Code Isolation_:It is unlikely to do anything substantial
> > in OFBiz without touching the core repositories. Compare this with
> > moqui or some other web frameworks like say Ruby-On-Rails or django
> > and you will see that you can work independently of the development
> > path in these frameworks. This makes upgrades quite painfulwith
> > patches and manual interventions. Some in the community even
> > introduced a whole patching framework in the past.
>
> We do not have this experience, at least not as much as you seem to
> have. Customer projects are developed completely in plugins, using the
> framework and avoiding changes there. There ARE some changes tagged as
> mods but not too much to be a real pain during a vendor update. Some
> mods are also contributed back to OFBiz so these mods disappear during
> the next vendor import.
>
> We are not directly using the original framework but rather use the
> original as an upstream branch for our own projects. This makes it easy
> to track changes and merge them into our codebase.
>
> It would be helpful to have some specific examples on where you are
> having those painful changes.
>
> >
> > _2. All or nothing:_In OFBiz, if you want to use the system, either
> > take the whole thing or take nothing. What if I don't want any user
> > interface for example, and just want to utilize the entities and
> > services?
>
> We are running projects which do not use the original user interface or
> are being headless / just REST interfaces. You are not forced to use the
> UI and you can switch off the web apps, do we really have a problem there?
>
> >
> > _3. Limited Re-usability:_Because of the way current components are
> > designed, it's difficult to take advantage of components in other
> > components. Sharing artifacts is pretty much limited to entities and
> > services. Even then they're not very shareable because they're
> > designed in a non-usable way for a certain context and certain
> > settings and database parameters.*
>
> Datamodel and services, the core business artifacts, are shareable and
> in my view that's what's needed mostly. Can you be more specific what's
> missing or designed in a way that it is not (re)usable ?
>
>
> > *
> >
> > _4. Fragmented Actions:_**Instead of having unified XML actions, you
> > have fragmentation across different contexts (screens, services,
> > SECAs, etc). Furthermore, you cannot mix XML actions with groovy or
> > other artifacts except by calling into external scripts and things
> > like that. In comparison to moqui which is more flexible in XML
> > actions I find it restrictive to work with OFBiz actions and it's hard
> > to review, debug, or modify. I'm listing something below as an example
> > for comparison [1]*
>
> I will have to investigate more how Moqui does those things. I'm not
> sure what the example shows. Is it part of the controller logic?
>
> Generally, I think that it is a strength to have these tasks separated
> in an MVC pattern and not being able to mix controller logic with
> business logic.
>
> When it comes to debugging: how do you debug the shown XML code?
>
>
> > *
> >
> > _5. Limited Routing capability:_**Routing is very minimal
> > (controller.xml). It cannot contain business logic and is limited
> > either to chaining requests or rendering screens. Sophisticated
> > routing logic is important for making rich systems that react to
> > differing circumstances.
> > **
>
> See above: the controller part should NOT contain too much business
> logic. If the business logic would be mixed with the controller logic,
> how would you then make the business logic more shareable (see point 3
> above)?
>
> Maybe I'm missing something and some examples would help me understand
> the point.
>
>
> >
> > _6. Core framework difficult to change:_**OFBiz core was written
> > a/long/time ago, and back then you didn't have many libraries for
> > caching, transactions, DB pooling, Concurrency stuff, and others, so a
> > lot was hand crafted, and we have a lots of interconnected API that is
> > difficult to change, and most of these APIs depend on static util
> > files that just kept piling and piling over the years. Where is the
> > service engine implementation exactly? What about the entity engine?
> > What about widgets? They are all over the place! Pieces for XML
> > parsing, others for DOM models, and so on. The code is scattered and
> > fixing it means going through a maze of many classes that are
> > interconnected and all referencing each other. It's also hard to
> > identify the signature or figure out the "big picture" since you have
> > essentially minimal interfaces and no clear public API that you can
> > refactor against. Pretty much the whole thing is public API**[2].
> > **
>
> That's a big field and I think we should break this into subtopics to
> discuss. Can you be more specific, for me this is too broad to discuss.
> What is the real problem, what would be a better solution, ...?
>
> In general, a framework like OFBiz IS complex, it will never be easy to
> read/understand. And of course, there is code which should be refactored
> and better maintained (deprecation etc.). It would maybe be a good start
> to identify those and do refactorings there first to reduce complexity
> and pave the path for bigger refactorings?
>
> From my experience, e.g. the caching capabilities of OFBiz are a big
> strength and to change those parts, we need some really good proof of
> concepts to check if other solutions are really better. I've had several
> bad experiences with, for example, object relational mapping frameworks
> (performance, complexity) and I'm still a fan of the entity engine.
>
>
> >
> > _7. Web-apps design problem__:_ A big problem (in my opinion) in the
> > design is around having multiple web-apps, one for each component (or
> > more). This interrupts the flow of the app and makes it difficult to
> > come up with smooth solutions that work across the various screens and
> > instead we need to pass a key between the components (and face other
> > problems around session management). This restricts you in what you
> > can offer as a user experience.*
> > *
>
> Do you have a solution in mind to solve this and also keep those
> applications separated (see above)? Single Sign On?
>
>
> >
> > _8. Problematic UI design__:_ Another/big/problem (in my opinion) is
> > having the UI show everything (reference implementation). I never
> > understood the value of showing/everything/! If I want to understand
> > the data model, I will just look at the data model. The purpose of the
> > UI is not to show everything, but to implement specific useful
> > use-cases. The reference implementation we have is not doing that, and
> > not offering something very useful, instead it is showing everything!
> > To improve this, we need to switch from data-model based interface
> > (party, order, etc ...) to use-case based interface (project
> > management, e-commerce, ERP, CRM, etc ...)*
> > *
>
> That's indeed a big issue and part of the discussions with every new
> user who wants to utilize the existing web apps. On one hand it is great
> to be able to show the potential of the ecosystem, on the other hand
> it's too complex, especially for new users.
>
> IMO this is not so much a problem of the framework in general, which
> does not prevent us from developing better UI and workflows. Personally,
> I see OFBiz as an ERP framework which has to be customized to match the
> target users's needs and not so much as an ready-to-use ERP.
>
> But yes, this can and should be improved with additional web apps or
> even ONE web app which tightly integrates all core applications into a
> one-stop ERP solution which is more user-friendly than the base
> applications. On the other hand, this will make separation more
> difficult...
>
>
> >
> > _9. Limited client rendering:_Because it is difficult to refactor the
> > framework, it's hard to pin-point where to go exactly to refactor all
> > the code needed to support more client-rendering technology (react,
> > vue, angular, etc ...) and make it interact smoothly with the back-end
> > (drop-downs, complex forms, etc ...). A lot of work needs to happen
> > both to the /themes and to /framework to allow and provide more
> > flexibility (also moving things to DB). If this step is not done you
> > will be stuck with your UI capabilities and won't catch up (unless you
> > develop your own UI which makes the UI of the community redundant and
> > you let go of the whole widget system anyway).*
> > *
>
> I guess it would help a lot if we have REST integrated into the core
> framework and expose a proper REST API. For the HTML based web UI, I
> have not much to change as the widget/freemarker templating gives us
> much flexibility.
>
> What do you miss exactly / in which direction do you want to do changes?
>
>
> >
> > _10. Security is difficult to implement:_One of my favorite features
> > in moqui is having security baked in the artifacts themselves to avoid
> > sprinkling all screens / services with security logic. I never even
> > worry about writing security related stuff until way close to the end
> > of delivering the project. So there needs to be a whole security layer
> > that sits below all the artifacts such that we don't even need to
> > write security code in the various screens and services and around
> > certain entities.*
> > *
>
> Can you elaborate more on how you want to achieve fine grained
> permissions for screens and services if not tied to the screen/service
> configurations (having in mind that they should be reusable)?
>
>
> >
> > _11. Limited Integration Solutions__:_ I know we have now a REST
> > component, but this is not the same as being a core feature of the
> > framework (essential for any modern web framework). Why should it be
> > tightly integrated? Many reasons: security, localization, mapping to
> > service engine, mapping to entities, etc ...*
> > *
>
> I'm currently working on the rest-api plugin and I agree that it would
> be great to have it in the framework instead of being a plugin.
>
>
> > _12. No Specialization:_Because of the big repository, and having
> > everything point to everything, it's hard to specialize. Now compare
> > this to the approach we have in our work which is easier to specialize
> > in. For example, I have a component for payment gateway integration,
> > another one for twilio (SMS integration), and I have application
> > components for each solution in its own repository (HR, insurance
> > stuff, many others) and each is living in its own git repo yet they
> > work together just fine and without touching one line of code in the
> > framework or even core components (despite depending on them and using
> > pieces of them). I can jump into new versions of the framework
> > everyday without breaking a sweat. When I work on component X, I'm
> > only thinking of that and nothing else. I don't have to worry about
> > changing something that would crash another component or bring the
> > system down (happened to me many times in the past).
>
> For our projects, we were going the other way: we have framework +
> plugins in the same repository instead of handling different
> repositories. I think it would be a real pain to split the project in
> dozens of different repositories only to bring them together to
> integrate during development or deployment again.
>
> Can you describe how you are handling those different repositories in an
> efficient way to get a better picture?
>
> As long as your work in a component/application/plugin and don't touch
> the framework code I also do not see why you should break something for
> other components? Of course, there ARE applications which are more
> fundamental than other (e.g. party, security) and if those are changed
> they might break other functionality. I see those applications belonging
> to the framework more than being independent applications.
>
> I agree that we should try to identify false or unnecessary dependencies
> though.
>
>
> >
> >
> > Solutions
> >
> > Whatever solutions we come up with I think they're going to be
> > difficult and would require community buy-in and support. I'm breaking
> > this down into things to keep, update, add and remove as follows:
> >
> >
> > Keep it
> >
> > - XSD signatures (so entity definitions, service definitions, XML
> > actions though should expand)
> > - Basic constructs of what is a component (with basic directory
> > structure)
> > - Data model
> > - Existing services (core data-model services)
> > - Java / groovy code that doesn't use core framework constructs or
> > goes too deep into the internals.
> > - Most used and useful APIs (Making queries, calling services, etc
> > ...) although it should all be converted to interfaces with separate
> > implementations where it's lacking.
> >
>
> Agree to all above.
>
>
> >
> > Improve It
> >
> > - Break the project to smaller git repositories.
> > - framework
> > - data model
> > - services
> > - common UI
> > - use-case applications (ERP, CRM, e-commerce, etc ...)
> > - plugins
> > - entire framework directory needs refurbishment and big chunks
> > probably require a delete / rewrite
> > - the web architecture (get rid of web-apps)
> > - the tests (lots of them need to be deleted anyway)
> > - Routing needs to be programmatic with ability to inject code.
> >
> >
> > Add It
> >
> > - Configuration override mechanism to remove the need to touch the
> > core. Overrides include everything including routing, component
> > registration, etc ...
> > - REST integration into the core
> > - Artifact based security that maps users / groups to artifacts
> > without the need to inject security code into every screen
> > - A super-set of XML actions that can be applied in all contexts and
> > can embed groovy code
> > - New, simple use-case based user interface. It needs to be designed
> > such that it is easy to extend to avoid the all-or-nothing UI that
> > currently exists.
> >
> >
> > Remove It / Deprecate It / Overwrite It
> >
> > - Most of the existing UI (needs to be replaced with simpler use-case
> > based interface)
> > - Lots of the core framework code (needs major rewrites, needs
> > interfaces, import libraries for things like transactions, security,
> > concurrency, etc ...)
> > - Probably some of the existing plugins might need to be deprecated.
>
> See my questions above for some of the topics. I will add more details
> and my point-of-view once we've split up the topics.
>
> The overall change would be huge and we should paint a clear picture of
> what we want to achieve and which way to go in detail.
>
> We'll have to prioritize with a good cost/effort consideration and do
> implementations step-by-step with a proper migration plan. Maybe having
> feature branches for a longer time to be able to put out releases if the
> development gets stuck or lasts longer.
>
>
> >
> >
> > Quality Control
> >
> > Investing in such a huge effort requires that we change the way we
> > view the project after / while work is being completed. Code should be
> > approached with greater care and a lot of emphasis should be put on
> > high quality, readability, avoiding anti-patterns, enforcing good
> > patterns and best-practices. Some of the daily workflows need to
> > improve / change, more thorough auditing and peer-reviews, and so on.
>
> A big +1 for quality control. This will be a real challenge for the
> project as it requires dedication and enough ressources for review.
>
>
> >
> >
> > Tradeoffs
> >
> > It goes without saying that the above is a _lot_ of work and
> > commitment and requires a lot of effort. That's why I hesitated in
> > starting this thread, but I thought maybe showing the tradeoffs would
> > make it easier to weigh things out and decide on a direction to push.
> >
> >
> > Pros
> >
> > - Pretty much most of the above mentioned problems will be resolved.
> > - The framework would be resilient and future proof.
> > - We will have more options and capabilities in the future, it would
> > be easier to incorporate new technologies as they emerge
> > - Development would go faster
> > - Technical Debt will be reduced
> >
> >
> > Cons
> >
> > - Huge (maybe too huge) upfront costs in time and effort to make the
> > switch
> > - Extra effort needed to ensure a smooth upgrade path for existing
> > users / adopters / system integrators. A whole layer of code needs to
> > be written to ensure the ability to upgrade without massive rewrites.
> > - Risk of failure, exhaustion, inability to push through to the end
> > - Inconvenience to those with existing systems / solutions
> >
> >
> > Summary
> >
> > Change is the only constant. Inability to adapt to the future means
> > not surviving. I'm not confident what is the best change possible or
> > at what pace, but I do know that lack of change is a death sentence to
> > any project. Maybe it's time to reflect and think of where we'd like
> > to go and this thread is sort of a brain-dump of possible ideas.
> >
> > I look forward to hearing your thoughts.
>
> Thanks, Taher, for the thoughtful write-up of your view on the project
> and the fields of change you see to be tackled. I do not agree with all
> of them and I have a sensible view on cost/effort, thorough planning and
> the need to always have existing users/projects in mind. But I hope that
> we/the project will find a way to constantly work on improvements.
>
> Again, I would propose to split up the most important topics into
> threads with accompanying documentation in the wiki to help avoid
> confusion.
>
> I am curious what others have to say and if we can activate enough
> interest to improve.
>
>
> >
> > Cheers,
> >
> > Taher Alkhateeb
> >
> >
> > [1] Example mixed XML actions and groovy from moqui
> >
> > <if condition="settlementTerm?.orderPmtServiceRegisterId">
> > <then>
> > <service-call name="update#mantle.order.OrderPart"
> > in-map="[orderId:orderId,
> > orderPartSeqId:orderPartSeqId, settlementTermId:settlementTermId]"/>
> >
> > <entity-find-one entity-name="moqui.service.ServiceRegister"
> > value-field="serviceRegister">
> > <field-map field-name="serviceRegisterId"
> > from="settlementTerm.orderPmtServiceRegisterId"/>
> > </entity-find-one>
> > <script><![CDATA[
> > try {
> >
> ec.service.sync().name(serviceRegister.serviceName).parameters(context).softValidate(true).disableAuthz().call()
> > if (ec.message.hasError()) return
> > } catch (Throwable t) {
> > ec.logger.log(300, "Error calling auto order payments
> > service for SettlementTerm ${settlementTermId}", t)
> > }
> > ]]></script>
> > </then>
> > <else>
> > <service-call
> > name="mantle.account.PaymentServices.create#Payment" out-map="context"
> > in-map="context + [amount:amount,
> > amountUomId:amountUomId, fromPartyId:fromPartyId, toPartyId:toPartyId]"/>
> > </else>
> > </if>
> >
> > [2] https://ci.apache.org/projects/ofbiz/site/trunk/javadocs/
> >
>
Re: OFBiz re-architecture thoughts
Posted by Divesh Dutta <di...@hotwaxsystems.com>.
I came across this old thread where we discussed OFBiz re-architecture.
From this thread, I conclude that Taher proposed many things; however, his core
idea was that upgrading OFBiz should be simple and predictable, which can
only happen if application code is completely isolated from the framework.
In many modern frameworks like Spring Boot, Ruby on Rails, and Django,
developers never modify the framework itself. The framework is treated as a
dependency, while all project-specific logic lives in separate application
modules. Because of this separation, when the framework releases a new
version, teams simply upgrade the dependency and their application
continues to work without manual merges or patches.
This is what Deepak also proposed in another thread: separating the
framework from applications.
And this time, the argument remains the same: in many real Apache OFBiz
implementations, developers end up modifying code inside framework/,
applications/, or shared core components. This creates tight coupling
between the project and the framework, making upgrades difficult because
every change must be manually reconciled with the new version.
So Taher and Deepak's proposal is to redesign OFBiz so the framework
remains untouched and projects are built as independent modules or
repositories that depend on it. With that structure, upgrading OFBiz would
become as simple as pulling the latest framework version, while application
code remains stable and unaffected.
Taher you didn't gain enough momentum 4 years ago, however, the community
has raised this point again , and I guess now is the right time to move
forward in this direction. I will appreciate your involvement in this
initiative.
Thanks
--
Divesh Dutta
On Tue, Jan 25, 2022 at 8:00 PM Taher Alkhateeb <ta...@pythys.com.invalid>
wrote:
> Thank you everyone for your kind feedback and sharing your thoughts on
> this initiative.
>
> I didn't get enough feedback or momentum to give me the impression that
> this can be a community initiative. I think I made the discussion too
> early before coming up with a PoC. So I will attempt something privately
> and come back perhaps when it is in a more ready state and see if it
> garners any momentum. IF people are interested in collaborating, then
> perhaps you can approach me and we can team up.
>
> Thank you again.
>
> On 1/9/22 11:16, Jacques Le Roux wrote:
> > Hi,
> >
> > If we want to improve OFBiz w/o changing too much the current OFBiz
> > why not create an official OFBiz2 fork and work to this goal there?
> >
> > Disclaimer: I don't think of doing something like that, if people are
> > ready for it, why not?
> >
> > Jacques
> >
> > Le 28/12/2021 à 18:18, Sakthivel Vellingiri a écrit :
> >> Thanks Taher and Michael for detailed insights and Sathish for
> >> sharing your
> >> thoughts on upgrades.
> >>
> >> Just to introduce myself, we are using OFBiz for little over 7 years
> >> for a
> >> Life Sciences Product, and we consider ourselves as end user and haven't
> >> inclined to make hardcore changes to the framework as yet.
> >>
> >> 1. Lack of Code Isolation_:::
> >> Our approach has been as outlined by Michael, we have custom
> >> plugins
> >> for each application and in some cases custom application plugins for
> >> each
> >> customer which are overrides and extensions of the OOTB OFBiz
> >> application
> >> plugins and we more or less leave the framework untouched except for
> >> a few
> >> issues we have discovered, and we are guilty of not contributing the fix
> >> back to OFBIZ and hence maintaining in our own branch, but without
> >> taking
> >> an inventory now, my recollection is that the changes to the
> >> framework is
> >> minimal. we merge the changes from our customer application plugin
> >> branch
> >> to *our* common application plugin branch that way *our* common
> >> application
> >> plugin branch at a given point in time has most of *our* custom
> >> application
> >> plugin changes.
> >>
> >> 2. Rest Interface:::
> >> We are currently using HTML based web UI and relying on
> >> widget/freemarker templating for the changes. We are inclined to client
> >> rendering using Vue Js, and having Rest API integrated in the core
> >> framework would be a huge plus. but we are open to using Rest API as a
> >> plugin to begin with, I know there has been some communication around
> >> Rest
> >> based webservice api in the email thread, I have to admit, we haven't
> >> explored that all that much yet. But if a Rest API plugin is made
> >> available, we will be happy to utilize that and share feedback.
> >>
> >> 3. UI Screens:::
> >> As experienced by new users and outlined in this thread, we end up
> >> customizing the UI to minimize the field display to our user needs
> >> either
> >> by making changes in minilang or ftl, Perhaps it might be good idea
> >> to have
> >> database tables to control what is rendered in the UI and perhaps
> >> rendering
> >> based on database configuration can include support for role/permission
> >> based rendering to show different fields for different
> >> roles/SecurityPermission.
> >>
> >> 4. Webapps :::
> >> As of right now, we switch off the webapps that we do not need
> >> currently, At less frequent times we see application logouts when
> >> switching
> >> from one webapp to another, i'm not sure if this caused by having
> >> different
> >> plugins as different webapps and passing the session id between them
> >> and if
> >> this be solved by having single webapp, but i do not understand enough
> >> about the merits of having a single webapp at this time to comment
> >> more on
> >> this.
> >>
> >> I do not have strong opinions about the other points other than what is
> >> discussed already and i will skip responding to them,
> >>
> >> In my opinion replacing OFBiz framework code with established open
> >> source
> >> which is purpose-built for a specific use case like Apache Shiro for
> >> Security, eHCache for caching could bring down OFBiz framework code and
> >> thereby maintenance of OFBiz, framework however i also recognize and
> >> reflect the thoughts outlined in this thread that we need to be very
> >> careful about bringing such hardcore changes with careful planning &
> >> quality control not to disrupt the existing implementations.
> >>
> >> Thanks to everyone sharing their view points and starting a
> >> conversation in
> >> lieu of the future of OFBiz.
> >>
> >> just my 2 cents from the end user perspective.
> >>
> >> regards
> >> Sakthi
> >>
> >>
> >> On Mon, Dec 27, 2021 at 1:09 PM Taher Alkhateeb
> >> <ta...@pythys.com.invalid>
> >> wrote:
> >>
> >>> Hi Michael,
> >>>
> >>> Thank you so much for spending your time reading and sifting through
> >>> everything, I really appreciate and value your feedback.
> >>>
> >>> I need to highlight that we deployed mixed and somewhat crazy solutions
> >>> including multiple mobile apps, live video / chat streaming systems and
> >>> other projects that are booming in Kuwait and exploding both in user
> >>> bases and technical requirements which is why we cannot operate some of
> >>> these projects the way we used to in the past.
> >>>
> >>> Anyway I think I'm not going to elaborate too much as we wait for more
> >>> opinions from the community to see if this is worth pushing. However, I
> >>> want to reply to a few points you raised:
> >>>
> >>> - There is a huge difference between _few_ modifications, and ZERO
> >>> modifications to the framework or core components. This way you don't
> >>> even need to freeze a version of OFBiz to develop against at all!
> >>> - Breaking to multiple git repositories means having something like
> >>> ./gradlew gitPullAll to update and you're done. Nothing breaks, no
> >>> surprises. You have to try it to believe it :) I have client projects
> >>> that have 15 git repos and I can upgrade with my eyes closed. Also
> >>> there
> >>> is always entanglement that you don't feel or care for until you
> >>> actually cut the repo into pieces and then realize we have too much
> >>> entanglement. Having many small git repositories is in fact the most
> >>> enjoyable part of how we're writing software now and I can assign
> >>> different modules to different teams and they work independent of each
> >>> other. It's just a totally different ballgame.
> >>> - All or nothing in terms of code means you have to onboard a lot of
> >>> code that you don't need. I'm not saying you cannot it turn it off, I'm
> >>> saying it should be opt-in, not opt-out. All this code is potential
> >>> maintenance, bugs, security, and so on that you shouldn't carry on if
> >>> you don't want it. This makes things the messiest in terms of seed /
> >>> demo data but also in other entanglements.
> >>> - Yes entities and services are re-usable but you're limited to only
> >>> that. Powerful re-usability is applied to everything (configurations,
> >>> routers, widgets, scripts, etc ...). What is mostly re-usable in OFBiz
> >>> is entities and services, the others are mostly "usable" not
> >>> "reusable".
> >>> It would take too long to explain and provide examples so I'll delay
> >>> this part.
> >>> - It's so awesome to use groovy exactly where you need it, without
> >>> having to jump into yet another file to only write 2 lines of code. I
> >>> have a service that needs 5-6 lines of code, why do I have to point to
> >>> an external script. Put the implementation RIGHT THERE next to the
> >>> definition of your service for example, the same is true everywhere you
> >>> need XML actions. So scripting in lacking in XML actions, and also they
> >>> are inconsistent (works differently in different places).
> >>> - Mixing XML with groovy is just part of the flexibility and having
> >>> your
> >>> code where you need it. Even debugging XML code improves by putting
> >>> groovy snippets above / below to investigate the context. Nobody would
> >>> want to do that by opening a new file just to print something from the
> >>> context for example.
> >>> - Services are pure business logic, controller is routing logic.
> >>> Routing
> >>> logic does not mean NO logic, it means logic that is focused on
> >>> coordination and decisions on where to go and what to do and putting
> >>> that logic in services is actually mixing layers and making the code
> >>> harder to understand / debug. I cannot show you how severely limited
> >>> routing logic is in OFBiz is without showing you 5 / 6 examples of
> >>> powerful things we've done that we simply can not do in OFBiz.
> >>> - We should IMHO by default have ONLY one web-app with the ability to
> >>> add more in an "opt-in" fashion. However, this means we need to be able
> >>> to break the screen definitions and routes, routing needs a major
> >>> change
> >>> to achieve this objective. The controller.xml would simply not cut it.
> >>> Again I need examples to really highlight what I mean and I might do it
> >>> later.
> >>> - It is not enough to integrate REST to be able to have dynamic
> >>> libraries integrated with OFBiz if you want that to work with your
> >>> widgets and other constructs. You also need multiple rendering modes
> >>> that allow you to select from a variety of different technologies /
> >>> options with different macros and different theming constructs. Again I
> >>> cannot say more without examples to illustrate clearly.
> >>> - Finally, many of the changes that I'm proposing require substantial
> >>> changes to the framework and redesigning it. Well we cannot do that
> >>> without first refactoring it, which means we have to get rid of /
> >>> update
> >>> lots of code. which means we need to start using libraries to control
> >>> the large size of that thing. It's a chain, one thing pulling another
> >>> thing, which makes it inevitable that a major shakeup of the core
> >>> framework is necessary. I'm not proposing to change it just to change
> >>> it. I'm proposing changing it because without that many of the things
> >>> I'm proposing simply won't happen.
> >>>
> >>> I can understand why you might not agree with some of the points. In a
> >>> way it's kind of hard to imagine a different way of doing things when
> >>> we're used to working a certain way, but once we get exposed we realize
> >>> that OH .. I've been torturing myself :). However, I do so your
> >>> very-good point on being conservative because there is substantial risk
> >>> in doing everything that I suggest.
> >>>
> >>> Anyway opening this to the rest of the community and thank you in
> >>> advance for sharing your thoughts.
> >>>
> >>> Cheers,
> >>>
> >>> Taher Alkhateeb
> >>>
> >>> On 12/27/21 17:20, Michael Brohl wrote:
> >>>> Hi Taher,
> >>>>
> >>>> thanks for bringing this up! I will try to briefly explain my views on
> >>>> the raised topics.
> >>>>
> >>>> My perspective is from nearly 20 years of intensive work with OFBiz,
> >>>> both for internal and external projects. Those projects have/had a
> >>>> volume reaching from about 50 to over 1000 project days and we have
> >>>> active customer projects which are running for almost 20 years now,
> >>>> still being developed further with the business. OFBiz often became a
> >>>> crucial part of the business ecosystem of those users. It is also the
> >>>> central toolbox for our services around development of web based
> >>>> software solutions and system integration in our own company since 7
> >>>> years now. I consider myself both a user of OFBiz and a service
> >>>> provider/integrator.
> >>>>
> >>>> After so many years of working with OFBiz if have learned that - from
> >>>> a business users perspective - the functionality and integration
> >>>> aspects along with the powerful datamodel, services etc. as well as
> >>>> flexibility, performance and scalability is way more important than
> >>>> detailed technical aspects (in the sense of “modern” or “newest
> >>>> technology”).
> >>>>
> >>>> As a service provider for OFBiz, we are feeling responsible to provide
> >>>> a reliable, constant and upgradeable software ecosystem which is why
> >>>> we are on a more "conservative" path regarding big or long running
> >>>> changes in the project. I have seen several new, ambititious
> >>>> contributors who wanted to change everything, using the newest
> >>>> frameworks and tools around etc., but often did not have an extensive
> >>>> background of real world projects or who are not able to explain why
> >>>> they want to use other tools and mechanisms and which user problems
> >>>> they really want to solve.
> >>>>
> >>>> Most of these people are not active anymore or unfinished solutions
> >>>> are put into the codebase which is another reason why I am really
> >>>> careful when it comes to massive changes. I think the project needs
> >>>> good planning, a good migration path and constant support when those
> >>>> changes are decided to take place.
> >>>>
> >>>> All in all, I do not feel such a great pain as you seem to feel but
> >>>> there is definetely much room for improvement, for sure.
> >>>>
> >>>> I will try to give my views on the different topics inline. They are
> >>>> brief and need/deserve more in-depth preparation from my side which I
> >>>> cannot spare at the moment.
> >>>>
> >>>> For further discussions I recommend to split those topics in different
> >>>> threads because they would become unreadable soon with more
> >>>> participants. Also writing up the outcome of the discussions and the
> >>>> further roadmap in the wiki might help.
> >>>>
> >>>> Let's see how it works.
> >>>>
> >>>> Michael Brohl
> >>>>
> >>>> ecomify GmbH - www.ecomify.de
> >>>>
> >>>>
> >>>> Am 22.12.21 um 18:23 schrieb Taher Alkhateeb:
> >>>>> Problems
> >>>> ...
> >>>>> _1. Lack of Code Isolation_:It is unlikely to do anything substantial
> >>>>> in OFBiz without touching the core repositories. Compare this with
> >>>>> moqui or some other web frameworks like say Ruby-On-Rails or django
> >>>>> and you will see that you can work independently of the development
> >>>>> path in these frameworks. This makes upgrades quite painfulwith
> >>>>> patches and manual interventions. Some in the community even
> >>>>> introduced a whole patching framework in the past.
> >>>> We do not have this experience, at least not as much as you seem to
> >>>> have. Customer projects are developed completely in plugins, using the
> >>>> framework and avoiding changes there. There ARE some changes tagged as
> >>>> mods but not too much to be a real pain during a vendor update. Some
> >>>> mods are also contributed back to OFBiz so these mods disappear during
> >>>> the next vendor import.
> >>>>
> >>>> We are not directly using the original framework but rather use the
> >>>> original as an upstream branch for our own projects. This makes it
> >>>> easy to track changes and merge them into our codebase.
> >>>>
> >>>> It would be helpful to have some specific examples on where you are
> >>>> having those painful changes.
> >>>>
> >>>>> _2. All or nothing:_In OFBiz, if you want to use the system, either
> >>>>> take the whole thing or take nothing. What if I don't want any user
> >>>>> interface for example, and just want to utilize the entities and
> >>>>> services?
> >>>> We are running projects which do not use the original user interface
> >>>> or are being headless / just REST interfaces. You are not forced to
> >>>> use the UI and you can switch off the web apps, do we really have a
> >>>> problem there?
> >>>>
> >>>>> _3. Limited Re-usability:_Because of the way current components are
> >>>>> designed, it's difficult to take advantage of components in other
> >>>>> components. Sharing artifacts is pretty much limited to entities and
> >>>>> services. Even then they're not very shareable because they're
> >>>>> designed in a non-usable way for a certain context and certain
> >>>>> settings and database parameters.*
> >>>> Datamodel and services, the core business artifacts, are shareable and
> >>>> in my view that's what's needed mostly. Can you be more specific
> >>>> what's missing or designed in a way that it is not (re)usable ?
> >>>>
> >>>>
> >>>>> *
> >>>>>
> >>>>> _4. Fragmented Actions:_**Instead of having unified XML actions, you
> >>>>> have fragmentation across different contexts (screens, services,
> >>>>> SECAs, etc). Furthermore, you cannot mix XML actions with groovy or
> >>>>> other artifacts except by calling into external scripts and things
> >>>>> like that. In comparison to moqui which is more flexible in XML
> >>>>> actions I find it restrictive to work with OFBiz actions and it's
> >>>>> hard to review, debug, or modify. I'm listing something below as an
> >>>>> example for comparison [1]*
> >>>> I will have to investigate more how Moqui does those things. I'm not
> >>>> sure what the example shows. Is it part of the controller logic?
> >>>>
> >>>> Generally, I think that it is a strength to have these tasks separated
> >>>> in an MVC pattern and not being able to mix controller logic with
> >>>> business logic.
> >>>>
> >>>> When it comes to debugging: how do you debug the shown XML code?
> >>>>
> >>>>
> >>>>> *
> >>>>>
> >>>>> _5. Limited Routing capability:_**Routing is very minimal
> >>>>> (controller.xml). It cannot contain business logic and is limited
> >>>>> either to chaining requests or rendering screens. Sophisticated
> >>>>> routing logic is important for making rich systems that react to
> >>>>> differing circumstances.
> >>>>> **
> >>>> See above: the controller part should NOT contain too much business
> >>>> logic. If the business logic would be mixed with the controller logic,
> >>>> how would you then make the business logic more shareable (see point 3
> >>>> above)?
> >>>>
> >>>> Maybe I'm missing something and some examples would help me understand
> >>>> the point.
> >>>>
> >>>>
> >>>>> _6. Core framework difficult to change:_**OFBiz core was written
> >>>>> a/long/time ago, and back then you didn't have many libraries for
> >>>>> caching, transactions, DB pooling, Concurrency stuff, and others, so
> >>>>> a lot was hand crafted, and we have a lots of interconnected API that
> >>>>> is difficult to change, and most of these APIs depend on static util
> >>>>> files that just kept piling and piling over the years. Where is the
> >>>>> service engine implementation exactly? What about the entity engine?
> >>>>> What about widgets? They are all over the place! Pieces for XML
> >>>>> parsing, others for DOM models, and so on. The code is scattered and
> >>>>> fixing it means going through a maze of many classes that are
> >>>>> interconnected and all referencing each other. It's also hard to
> >>>>> identify the signature or figure out the "big picture" since you have
> >>>>> essentially minimal interfaces and no clear public API that you can
> >>>>> refactor against. Pretty much the whole thing is public API**[2].
> >>>>> **
> >>>> That's a big field and I think we should break this into subtopics to
> >>>> discuss. Can you be more specific, for me this is too broad to
> >>>> discuss. What is the real problem, what would be a better solution,
> >>>> ...?
> >>>>
> >>>> In general, a framework like OFBiz IS complex, it will never be easy
> >>>> to read/understand. And of course, there is code which should be
> >>>> refactored and better maintained (deprecation etc.). It would maybe be
> >>>> a good start to identify those and do refactorings there first to
> >>>> reduce complexity and pave the path for bigger refactorings?
> >>>>
> >>>> From my experience, e.g. the caching capabilities of OFBiz are a big
> >>>> strength and to change those parts, we need some really good proof of
> >>>> concepts to check if other solutions are really better. I've had
> >>>> several bad experiences with, for example, object relational mapping
> >>>> frameworks (performance, complexity) and I'm still a fan of the entity
> >>>> engine.
> >>>>
> >>>>
> >>>>> _7. Web-apps design problem__:_ A big problem (in my opinion) in the
> >>>>> design is around having multiple web-apps, one for each component (or
> >>>>> more). This interrupts the flow of the app and makes it difficult to
> >>>>> come up with smooth solutions that work across the various screens
> >>>>> and instead we need to pass a key between the components (and face
> >>>>> other problems around session management). This restricts you in what
> >>>>> you can offer as a user experience.*
> >>>>> *
> >>>> Do you have a solution in mind to solve this and also keep those
> >>>> applications separated (see above)? Single Sign On?
> >>>>
> >>>>
> >>>>> _8. Problematic UI design__:_ Another/big/problem (in my opinion) is
> >>>>> having the UI show everything (reference implementation). I never
> >>>>> understood the value of showing/everything/! If I want to understand
> >>>>> the data model, I will just look at the data model. The purpose of
> >>>>> the UI is not to show everything, but to implement specific useful
> >>>>> use-cases. The reference implementation we have is not doing that,
> >>>>> and not offering something very useful, instead it is showing
> >>>>> everything! To improve this, we need to switch from data-model based
> >>>>> interface (party, order, etc ...) to use-case based interface
> >>>>> (project management, e-commerce, ERP, CRM, etc ...)*
> >>>>> *
> >>>> That's indeed a big issue and part of the discussions with every new
> >>>> user who wants to utilize the existing web apps. On one hand it is
> >>>> great to be able to show the potential of the ecosystem, on the other
> >>>> hand it's too complex, especially for new users.
> >>>>
> >>>> IMO this is not so much a problem of the framework in general, which
> >>>> does not prevent us from developing better UI and workflows.
> >>>> Personally, I see OFBiz as an ERP framework which has to be customized
> >>>> to match the target users's needs and not so much as an
> >>>> ready-to-use ERP.
> >>>>
> >>>> But yes, this can and should be improved with additional web apps or
> >>>> even ONE web app which tightly integrates all core applications into a
> >>>> one-stop ERP solution which is more user-friendly than the base
> >>>> applications. On the other hand, this will make separation more
> >>>> difficult...
> >>>>
> >>>>
> >>>>> _9. Limited client rendering:_Because it is difficult to refactor the
> >>>>> framework, it's hard to pin-point where to go exactly to refactor all
> >>>>> the code needed to support more client-rendering technology (react,
> >>>>> vue, angular, etc ...) and make it interact smoothly with the
> >>>>> back-end (drop-downs, complex forms, etc ...). A lot of work needs to
> >>>>> happen both to the /themes and to /framework to allow and provide
> >>>>> more flexibility (also moving things to DB). If this step is not done
> >>>>> you will be stuck with your UI capabilities and won't catch up
> >>>>> (unless you develop your own UI which makes the UI of the community
> >>>>> redundant and you let go of the whole widget system anyway).*
> >>>>> *
> >>>> I guess it would help a lot if we have REST integrated into the core
> >>>> framework and expose a proper REST API. For the HTML based web UI, I
> >>>> have not much to change as the widget/freemarker templating gives us
> >>>> much flexibility.
> >>>>
> >>>> What do you miss exactly / in which direction do you want to do
> >>>> changes?
> >>>>
> >>>>
> >>>>> _10. Security is difficult to implement:_One of my favorite features
> >>>>> in moqui is having security baked in the artifacts themselves to
> >>>>> avoid sprinkling all screens / services with security logic. I never
> >>>>> even worry about writing security related stuff until way close to
> >>>>> the end of delivering the project. So there needs to be a whole
> >>>>> security layer that sits below all the artifacts such that we don't
> >>>>> even need to write security code in the various screens and services
> >>>>> and around certain entities.*
> >>>>> *
> >>>> Can you elaborate more on how you want to achieve fine grained
> >>>> permissions for screens and services if not tied to the screen/service
> >>>> configurations (having in mind that they should be reusable)?
> >>>>
> >>>>
> >>>>> _11. Limited Integration Solutions__:_ I know we have now a REST
> >>>>> component, but this is not the same as being a core feature of the
> >>>>> framework (essential for any modern web framework). Why should it be
> >>>>> tightly integrated? Many reasons: security, localization, mapping to
> >>>>> service engine, mapping to entities, etc ...*
> >>>>> *
> >>>> I'm currently working on the rest-api plugin and I agree that it would
> >>>> be great to have it in the framework instead of being a plugin.
> >>>>
> >>>>
> >>>>> _12. No Specialization:_Because of the big repository, and having
> >>>>> everything point to everything, it's hard to specialize. Now compare
> >>>>> this to the approach we have in our work which is easier to
> >>>>> specialize in. For example, I have a component for payment gateway
> >>>>> integration, another one for twilio (SMS integration), and I have
> >>>>> application components for each solution in its own repository (HR,
> >>>>> insurance stuff, many others) and each is living in its own git repo
> >>>>> yet they work together just fine and without touching one line of
> >>>>> code in the framework or even core components (despite depending on
> >>>>> them and using pieces of them). I can jump into new versions of the
> >>>>> framework everyday without breaking a sweat. When I work on component
> >>>>> X, I'm only thinking of that and nothing else. I don't have to worry
> >>>>> about changing something that would crash another component or bring
> >>>>> the system down (happened to me many times in the past).
> >>>> For our projects, we were going the other way: we have framework +
> >>>> plugins in the same repository instead of handling different
> >>>> repositories. I think it would be a real pain to split the project in
> >>>> dozens of different repositories only to bring them together to
> >>>> integrate during development or deployment again.
> >>>>
> >>>> Can you describe how you are handling those different repositories in
> >>>> an efficient way to get a better picture?
> >>>>
> >>>> As long as your work in a component/application/plugin and don't touch
> >>>> the framework code I also do not see why you should break something
> >>>> for other components? Of course, there ARE applications which are more
> >>>> fundamental than other (e.g. party, security) and if those are changed
> >>>> they might break other functionality. I see those applications
> >>>> belonging to the framework more than being independent applications.
> >>>>
> >>>> I agree that we should try to identify false or unnecessary
> >>>> dependencies though.
> >>>>
> >>>>
> >>>>>
> >>>>> Solutions
> >>>>>
> >>>>> Whatever solutions we come up with I think they're going to be
> >>>>> difficult and would require community buy-in and support. I'm
> >>>>> breaking this down into things to keep, update, add and remove as
> >>>>> follows:
> >>>>>
> >>>>>
> >>>>> Keep it
> >>>>>
> >>>>> - XSD signatures (so entity definitions, service definitions, XML
> >>>>> actions though should expand)
> >>>>> - Basic constructs of what is a component (with basic directory
> >>>>> structure)
> >>>>> - Data model
> >>>>> - Existing services (core data-model services)
> >>>>> - Java / groovy code that doesn't use core framework constructs or
> >>>>> goes too deep into the internals.
> >>>>> - Most used and useful APIs (Making queries, calling services, etc
> >>>>> ...) although it should all be converted to interfaces with separate
> >>>>> implementations where it's lacking.
> >>>>>
> >>>> Agree to all above.
> >>>>
> >>>>
> >>>>> Improve It
> >>>>>
> >>>>> - Break the project to smaller git repositories.
> >>>>> - framework
> >>>>> - data model
> >>>>> - services
> >>>>> - common UI
> >>>>> - use-case applications (ERP, CRM, e-commerce, etc ...)
> >>>>> - plugins
> >>>>> - entire framework directory needs refurbishment and big chunks
> >>>>> probably require a delete / rewrite
> >>>>> - the web architecture (get rid of web-apps)
> >>>>> - the tests (lots of them need to be deleted anyway)
> >>>>> - Routing needs to be programmatic with ability to inject code.
> >>>>>
> >>>>>
> >>>>> Add It
> >>>>>
> >>>>> - Configuration override mechanism to remove the need to touch the
> >>>>> core. Overrides include everything including routing, component
> >>>>> registration, etc ...
> >>>>> - REST integration into the core
> >>>>> - Artifact based security that maps users / groups to artifacts
> >>>>> without the need to inject security code into every screen
> >>>>> - A super-set of XML actions that can be applied in all contexts and
> >>>>> can embed groovy code
> >>>>> - New, simple use-case based user interface. It needs to be designed
> >>>>> such that it is easy to extend to avoid the all-or-nothing UI that
> >>>>> currently exists.
> >>>>>
> >>>>>
> >>>>> Remove It / Deprecate It / Overwrite It
> >>>>>
> >>>>> - Most of the existing UI (needs to be replaced with simpler use-case
> >>>>> based interface)
> >>>>> - Lots of the core framework code (needs major rewrites, needs
> >>>>> interfaces, import libraries for things like transactions, security,
> >>>>> concurrency, etc ...)
> >>>>> - Probably some of the existing plugins might need to be deprecated.
> >>>> See my questions above for some of the topics. I will add more details
> >>>> and my point-of-view once we've split up the topics.
> >>>>
> >>>> The overall change would be huge and we should paint a clear picture
> >>>> of what we want to achieve and which way to go in detail.
> >>>>
> >>>> We'll have to prioritize with a good cost/effort consideration and do
> >>>> implementations step-by-step with a proper migration plan. Maybe
> >>>> having feature branches for a longer time to be able to put out
> >>>> releases if the development gets stuck or lasts longer.
> >>>>
> >>>>
> >>>>>
> >>>>> Quality Control
> >>>>>
> >>>>> Investing in such a huge effort requires that we change the way we
> >>>>> view the project after / while work is being completed. Code should
> >>>>> be approached with greater care and a lot of emphasis should be put
> >>>>> on high quality, readability, avoiding anti-patterns, enforcing good
> >>>>> patterns and best-practices. Some of the daily workflows need to
> >>>>> improve / change, more thorough auditing and peer-reviews, and so on.
> >>>> A big +1 for quality control. This will be a real challenge for the
> >>>> project as it requires dedication and enough ressources for review.
> >>>>
> >>>>
> >>>>>
> >>>>> Tradeoffs
> >>>>>
> >>>>> It goes without saying that the above is a _lot_ of work and
> >>>>> commitment and requires a lot of effort. That's why I hesitated in
> >>>>> starting this thread, but I thought maybe showing the tradeoffs would
> >>>>> make it easier to weigh things out and decide on a direction to push.
> >>>>>
> >>>>>
> >>>>> Pros
> >>>>>
> >>>>> - Pretty much most of the above mentioned problems will be resolved.
> >>>>> - The framework would be resilient and future proof.
> >>>>> - We will have more options and capabilities in the future, it would
> >>>>> be easier to incorporate new technologies as they emerge
> >>>>> - Development would go faster
> >>>>> - Technical Debt will be reduced
> >>>>>
> >>>>>
> >>>>> Cons
> >>>>>
> >>>>> - Huge (maybe too huge) upfront costs in time and effort to make the
> >>>>> switch
> >>>>> - Extra effort needed to ensure a smooth upgrade path for existing
> >>>>> users / adopters / system integrators. A whole layer of code needs to
> >>>>> be written to ensure the ability to upgrade without massive rewrites.
> >>>>> - Risk of failure, exhaustion, inability to push through to the end
> >>>>> - Inconvenience to those with existing systems / solutions
> >>>>>
> >>>>>
> >>>>> Summary
> >>>>>
> >>>>> Change is the only constant. Inability to adapt to the future means
> >>>>> not surviving. I'm not confident what is the best change possible or
> >>>>> at what pace, but I do know that lack of change is a death sentence
> >>>>> to any project. Maybe it's time to reflect and think of where we'd
> >>>>> like to go and this thread is sort of a brain-dump of possible ideas.
> >>>>>
> >>>>> I look forward to hearing your thoughts.
> >>>> Thanks, Taher, for the thoughtful write-up of your view on the project
> >>>> and the fields of change you see to be tackled. I do not agree with
> >>>> all of them and I have a sensible view on cost/effort, thorough
> >>>> planning and the need to always have existing users/projects in mind.
> >>>> But I hope that we/the project will find a way to constantly work on
> >>>> improvements.
> >>>>
> >>>> Again, I would propose to split up the most important topics into
> >>>> threads with accompanying documentation in the wiki to help avoid
> >>>> confusion.
> >>>>
> >>>> I am curious what others have to say and if we can activate enough
> >>>> interest to improve.
> >>>>
> >>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Taher Alkhateeb
> >>>>>
> >>>>>
> >>>>> [1] Example mixed XML actions and groovy from moqui
> >>>>>
> >>>>> <if condition="settlementTerm?.orderPmtServiceRegisterId">
> >>>>> <then>
> >>>>> <service-call name="update#mantle.order.OrderPart"
> >>>>> in-map="[orderId:orderId,
> >>>>> orderPartSeqId:orderPartSeqId, settlementTermId:settlementTermId]"/>
> >>>>>
> >>>>> <entity-find-one entity-name="moqui.service.ServiceRegister"
> >>>>> value-field="serviceRegister">
> >>>>> <field-map field-name="serviceRegisterId"
> >>>>> from="settlementTerm.orderPmtServiceRegisterId"/>
> >>>>> </entity-find-one>
> >>>>> <script><![CDATA[
> >>>>> try {
> >>>>>
> >>>
> ec.service.sync().name(serviceRegister.serviceName).parameters(context).softValidate(true).disableAuthz().call()
>
> >>>
> >>>
> >>>>> if (ec.message.hasError()) return
> >>>>> } catch (Throwable t) {
> >>>>> ec.logger.log(300, "Error calling auto order
> >>>>> payments
> >>>>> service for SettlementTerm ${settlementTermId}", t)
> >>>>> }
> >>>>> ]]></script>
> >>>>> </then>
> >>>>> <else>
> >>>>> <service-call
> >>>>> name="mantle.account.PaymentServices.create#Payment"
> >>>>> out-map="context"
> >>>>> in-map="context + [amount:amount,
> >>>>> amountUomId:amountUomId, fromPartyId:fromPartyId,
> >>>>> toPartyId:toPartyId]"/>
> >>>>> </else>
> >>>>> </if>
> >>>>>
> >>>>> [2] https://ci.apache.org/projects/ofbiz/site/trunk/javadocs/
> >>>>>
>
Re: OFBiz re-architecture thoughts
Posted by Jacques Le Roux <ja...@les7arts.com>.
Hi Nicola,
I think Moqui is near from that.
Though not totally started from OFBiz it's quite inspired from David's long experience with OFBiz
Jacques
Le 02/04/2022 à 19:29, Nicola Mazzoni a écrit :
> I think it would be very interesting if some community developer were interested in creating a * small * POC of a DDD project. Using the OFBiz
> Entity Engine by exploiting the DDD concepts and therefore the generation of entity objects, value objects, repositories ... We could take a
> "domain", then a small part of OFBiz data-model and experiment with this approach. Leaving aside the graphical interface part, focusing only on the
> core.
>
> What do you think?
Re: OFBiz re-architecture thoughts
Posted by Nicola Mazzoni <ni...@mpstyle.it>.
Hi Taher,
From my experience I have chosen OFBiz because the only "coherent"
framework with a data model and related services. Completely different
from all frameworks that are architectural only but remain an "empty
box" for building applications from scratch. I think there are many
developers who need a business application framework (with an underlying
data model) but they don't come close to OFBiz, I think because of the
complexity and being, paradoxically, too complete. For example, the zip
distribution already suggests an "all or nothing" use. Perhaps these
things do not bring new potential users of OFBiz closer.
I think that an OOP logic is perfectly compatible with the entity and
service engines, xml-dsl etc etc; indeed, they can be used as a basis to
create a higher level of abstraction, which through OOP makes it more
accessible.
With this I don't think of "changing" OFBiz for what it is, for the
backward compatibility and for the solidity of the solutions based on
OFBiz. It could be interesting, if there are interested community
members, to try a different path, with the solid foundations of OFBiz
but on a different paradigm. I don't know if this POC could be
interesting enough to become a sub-project or it could simply remain a
POC; but if anyone is interested it would be interesting to try!
Regards,
Nicola
On 03/04/22 19:12, Taher Alkhateeb wrote:
> Hi Nicola,
>
> Thank you for sharing your thoughts. It's good to have a practical
> perspective.
>
> If we try to maybe list what OFBiz offers as highest value (guessing
> on behalf of folks) I would think it would include:
>
> - The entity and service engines with their APIs
> - The data model represented by entities
> - The services that serve the data model
> - The XML DSL that abstracts away the underlying technologies
> (widgets, actions, entities, etc ...)
>
> So OFBiz in that sense is a radical departure from OO since it all
> depends on the abstractions of services (SAO) with an XML-based DSL to
> manage everything. Maybe this is what people like or want the most out
> of it.
>
> So whether or not we should attempt a PoC depends on whether people
> actually want that different architecture since a DDD perspective is a
> significant shift (especially with bounded contexts in an OO style)
> from what OFBiz offers.
>
> Maybe more opinions here might shed light.
>
> Cheers,
>
> Taher
>
> On 4/2/22 20:29, Nicola Mazzoni wrote:
>> Hi Taher,
>> I think it's a very interesting discussion. She made me think a lot.
>> I agree with your thoughts, but working as OFBiz "system integrator"
>> I fully agree with Michael.
>> OFBiz is not a ready-to-use ERP system but it is a fantastic
>> framework for developing custom ERP solutions. ERPs have a long, very
>> long life cycle and are less sensitive to technological innovations.
>> Often ERP and legacy are synonymous, while all the surrounding
>> applications change, the ERPs remain. My clients in Erp, as in the
>> "traditional" management of existing business processes, do not ask
>> for a "cutting-edge" system. Also working on the IBM mainframe I know
>> the subject, and I often suffer the consequences, not being able to
>> "move forward"
>>
>> Since the beginning of 2022 I have had to take over the design "from
>> scratch" of a business application and lead a small DevOps team to
>> follow the development. I've waited 3 months to reply to your email
>> to share this new experience with a quarter of feedback behind it.
>> The project is not a "complete" ERP, but it is part of it. I didn't
>> start from OFBiz, it was a solution with technological requirements
>> very far from OFBiz and too tied to an already existing datamodel.
>> But most of all, I wanted to experiment with a new (for me) approach.
>> Just as OFBiz is closely linked to Len Silverstone's fantastic
>> "Datamodel resource book", I wanted to find Eric Evans' "Domain
>> Driven Design" as a "literary reference".
>> I don't like ORMs and therefore a very light persistance framework
>> (like MyBatis) has been used, keeping the queries "handwritten" and
>> not generated by the system. The goal was to make a completely object
>> oriented, clean and simple system, not heavily tied to any existing
>> framework. I followed the DDD approach (not in a dogmatic way ...)
>> avoiding any third party libraries (excluding Jackson, Apache
>> Commons, Apache Commons Pool, Apache CXF). I could have used Spring /
>> Spring data; but I didn't. I have nothing against Spring, but I would
>> like to use only frameworks that I know well, otherwise it is easy to
>> become a victim. I would never want to tell a client "you know, my
>> program is broken because there's something under the hood that I
>> don't know that doesn't make it work anymore"
>>
>> In short? It was very interesting! The most significant effect (for
>> the development team / qa) was certainly the simplicity with which
>> the junior developers were able to be productive (having already the
>> object model done) by working on a clean and very readable code. In
>> general, the only drawback is a "structured" approach at the
>> beginning which certainly requires considerable design experience and
>> time. Precisely this, however, then allows an "agile" and very fast
>> approach in tests, in customer feedback, in reworks. I don't think
>> DDD (neither microservices, nor ORMs, nor noSql ...) is the silver
>> bullet that solves all problems. But we can intelligently take
>> advantage of these approaches without marrying them.
>>
>> I think it would be very interesting if some community developer were
>> interested in creating a * small * POC of a DDD project. Using the
>> OFBiz Entity Engine by exploiting the DDD concepts and therefore the
>> generation of entity objects, value objects, repositories ... We
>> could take a "domain", then a small part of OFBiz data-model and
>> experiment with this approach. Leaving aside the graphical interface
>> part, focusing only on the core.
>>
>> What do you think?
>>
>> Regards,
>>
>> Nicola
>>
>> On 01/03/22 04:56, Aditya Sharma wrote:
>>> Hi Taher,
>>>
>>> Thanks for this initiative!
>>>
>>> How about incubating it and adopting it as a subproject?
>>>
>>> Thanks and Regards,
>>> Aditya Sharma
>>>
>>> On Tue, Jan 25, 2022 at 10:28 PM Eugen Stan<eu...@netdava.com>
>>> wrote:
>>>
>>>> Hello Taher,
>>>>
>>>> Thank you for this initiative and sorry for not replying sooner.
>>>> I meant to reply for so long but life ...
>>>>
>>>> I think most / all agrees that change is needed in OFBiz but I
>>>> guess we
>>>> have different ideas on how to proceed.
>>>>
>>>> I don't think I can add a lot to what was already said.
>>>>
>>>> I do think keeping the current number of git repositories is good.
>>>> More will add overhead.
>>>> I do believe that ofbiz should provide stable extension points so
>>>> people
>>>> can deploy their own plugins and functionality.
>>>>
>>>> I'm also working on some ideas around OFBiz - I believe I can
>>>> re-implement OFBiz entity engine on top of relational algebra using
>>>> Calcite.
>>>>
>>>> More on this soon on the mailing list.
>>>>
>>>> Looking forward to seeing a PoC from you.
>>>> I hope we do get to improve OFBiz and make it even more appealing.
>>>>
>>>>
>>>> On 25.01.2022 16:29, Taher Alkhateeb wrote:
>>>>> Thank you everyone for your kind feedback and sharing your
>>>>> thoughts on
>>>>> this initiative.
>>>>>
>>>>> I didn't get enough feedback or momentum to give me the impression
>>>>> that
>>>>> this can be a community initiative. I think I made the discussion too
>>>>> early before coming up with a PoC. So I will attempt something
>>>>> privately
>>>>> and come back perhaps when it is in a more ready state and see if it
>>>>> garners any momentum. IF people are interested in collaborating, then
>>>>> perhaps you can approach me and we can team up.
>>>>>
>>>>> Thank you again.
>>>> Regards,
>>>>
>>>> --
>>>> Eugen Stan
>>>>
>>>> +40770 941 271 /https://www.netdava.com
--
*Nicola Mazzoni* | /CEO/CTO/
Mobile: +39 347 990 5529
*Mp Style srl*
Via Meucci, 37 | 41019 Limidi di Soliera (MO)
p.i / c.f IT03679300362
Re: OFBiz re-architecture thoughts
Posted by Taher Alkhateeb <ta...@pythys.com.INVALID>.
Hi Nicola,
Thank you for sharing your thoughts. It's good to have a practical
perspective.
If we try to maybe list what OFBiz offers as highest value (guessing on
behalf of folks) I would think it would include:
- The entity and service engines with their APIs
- The data model represented by entities
- The services that serve the data model
- The XML DSL that abstracts away the underlying technologies (widgets,
actions, entities, etc ...)
So OFBiz in that sense is a radical departure from OO since it all
depends on the abstractions of services (SAO) with an XML-based DSL to
manage everything. Maybe this is what people like or want the most out
of it.
So whether or not we should attempt a PoC depends on whether people
actually want that different architecture since a DDD perspective is a
significant shift (especially with bounded contexts in an OO style) from
what OFBiz offers.
Maybe more opinions here might shed light.
Cheers,
Taher
On 4/2/22 20:29, Nicola Mazzoni wrote:
> Hi Taher,
> I think it's a very interesting discussion. She made me think a lot. I
> agree with your thoughts, but working as OFBiz "system integrator" I
> fully agree with Michael.
> OFBiz is not a ready-to-use ERP system but it is a fantastic framework
> for developing custom ERP solutions. ERPs have a long, very long life
> cycle and are less sensitive to technological innovations. Often ERP
> and legacy are synonymous, while all the surrounding applications
> change, the ERPs remain. My clients in Erp, as in the "traditional"
> management of existing business processes, do not ask for a
> "cutting-edge" system. Also working on the IBM mainframe I know the
> subject, and I often suffer the consequences, not being able to "move
> forward"
>
> Since the beginning of 2022 I have had to take over the design "from
> scratch" of a business application and lead a small DevOps team to
> follow the development. I've waited 3 months to reply to your email to
> share this new experience with a quarter of feedback behind it. The
> project is not a "complete" ERP, but it is part of it. I didn't start
> from OFBiz, it was a solution with technological requirements very far
> from OFBiz and too tied to an already existing datamodel. But most of
> all, I wanted to experiment with a new (for me) approach. Just as
> OFBiz is closely linked to Len Silverstone's fantastic "Datamodel
> resource book", I wanted to find Eric Evans' "Domain Driven Design" as
> a "literary reference".
> I don't like ORMs and therefore a very light persistance framework
> (like MyBatis) has been used, keeping the queries "handwritten" and
> not generated by the system. The goal was to make a completely object
> oriented, clean and simple system, not heavily tied to any existing
> framework. I followed the DDD approach (not in a dogmatic way ...)
> avoiding any third party libraries (excluding Jackson, Apache Commons,
> Apache Commons Pool, Apache CXF). I could have used Spring / Spring
> data; but I didn't. I have nothing against Spring, but I would like to
> use only frameworks that I know well, otherwise it is easy to become a
> victim. I would never want to tell a client "you know, my program is
> broken because there's something under the hood that I don't know that
> doesn't make it work anymore"
>
> In short? It was very interesting! The most significant effect (for
> the development team / qa) was certainly the simplicity with which the
> junior developers were able to be productive (having already the
> object model done) by working on a clean and very readable code. In
> general, the only drawback is a "structured" approach at the beginning
> which certainly requires considerable design experience and time.
> Precisely this, however, then allows an "agile" and very fast approach
> in tests, in customer feedback, in reworks. I don't think DDD (neither
> microservices, nor ORMs, nor noSql ...) is the silver bullet that
> solves all problems. But we can intelligently take advantage of these
> approaches without marrying them.
>
> I think it would be very interesting if some community developer were
> interested in creating a * small * POC of a DDD project. Using the
> OFBiz Entity Engine by exploiting the DDD concepts and therefore the
> generation of entity objects, value objects, repositories ... We could
> take a "domain", then a small part of OFBiz data-model and experiment
> with this approach. Leaving aside the graphical interface part,
> focusing only on the core.
>
> What do you think?
>
> Regards,
>
> Nicola
>
> On 01/03/22 04:56, Aditya Sharma wrote:
>> Hi Taher,
>>
>> Thanks for this initiative!
>>
>> How about incubating it and adopting it as a subproject?
>>
>> Thanks and Regards,
>> Aditya Sharma
>>
>> On Tue, Jan 25, 2022 at 10:28 PM Eugen Stan<eu...@netdava.com>
>> wrote:
>>
>>> Hello Taher,
>>>
>>> Thank you for this initiative and sorry for not replying sooner.
>>> I meant to reply for so long but life ...
>>>
>>> I think most / all agrees that change is needed in OFBiz but I guess we
>>> have different ideas on how to proceed.
>>>
>>> I don't think I can add a lot to what was already said.
>>>
>>> I do think keeping the current number of git repositories is good.
>>> More will add overhead.
>>> I do believe that ofbiz should provide stable extension points so
>>> people
>>> can deploy their own plugins and functionality.
>>>
>>> I'm also working on some ideas around OFBiz - I believe I can
>>> re-implement OFBiz entity engine on top of relational algebra using
>>> Calcite.
>>>
>>> More on this soon on the mailing list.
>>>
>>> Looking forward to seeing a PoC from you.
>>> I hope we do get to improve OFBiz and make it even more appealing.
>>>
>>>
>>> On 25.01.2022 16:29, Taher Alkhateeb wrote:
>>>> Thank you everyone for your kind feedback and sharing your thoughts on
>>>> this initiative.
>>>>
>>>> I didn't get enough feedback or momentum to give me the impression
>>>> that
>>>> this can be a community initiative. I think I made the discussion too
>>>> early before coming up with a PoC. So I will attempt something
>>>> privately
>>>> and come back perhaps when it is in a more ready state and see if it
>>>> garners any momentum. IF people are interested in collaborating, then
>>>> perhaps you can approach me and we can team up.
>>>>
>>>> Thank you again.
>>> Regards,
>>>
>>> --
>>> Eugen Stan
>>>
>>> +40770 941 271 /https://www.netdava.com
Re: OFBiz re-architecture thoughts
Posted by Nicola Mazzoni <ni...@mpstyle.it>.
Hi Taher,
I think it's a very interesting discussion. She made me think a lot. I
agree with your thoughts, but working as OFBiz "system integrator" I
fully agree with Michael.
OFBiz is not a ready-to-use ERP system but it is a fantastic framework
for developing custom ERP solutions. ERPs have a long, very long life
cycle and are less sensitive to technological innovations. Often ERP and
legacy are synonymous, while all the surrounding applications change,
the ERPs remain. My clients in Erp, as in the "traditional" management
of existing business processes, do not ask for a "cutting-edge" system.
Also working on the IBM mainframe I know the subject, and I often suffer
the consequences, not being able to "move forward"
Since the beginning of 2022 I have had to take over the design "from
scratch" of a business application and lead a small DevOps team to
follow the development. I've waited 3 months to reply to your email to
share this new experience with a quarter of feedback behind it. The
project is not a "complete" ERP, but it is part of it. I didn't start
from OFBiz, it was a solution with technological requirements very far
from OFBiz and too tied to an already existing datamodel. But most of
all, I wanted to experiment with a new (for me) approach. Just as OFBiz
is closely linked to Len Silverstone's fantastic "Datamodel resource
book", I wanted to find Eric Evans' "Domain Driven Design" as a
"literary reference".
I don't like ORMs and therefore a very light persistance framework (like
MyBatis) has been used, keeping the queries "handwritten" and not
generated by the system. The goal was to make a completely object
oriented, clean and simple system, not heavily tied to any existing
framework. I followed the DDD approach (not in a dogmatic way ...)
avoiding any third party libraries (excluding Jackson, Apache Commons,
Apache Commons Pool, Apache CXF). I could have used Spring / Spring
data; but I didn't. I have nothing against Spring, but I would like to
use only frameworks that I know well, otherwise it is easy to become a
victim. I would never want to tell a client "you know, my program is
broken because there's something under the hood that I don't know that
doesn't make it work anymore"
In short? It was very interesting! The most significant effect (for the
development team / qa) was certainly the simplicity with which the
junior developers were able to be productive (having already the object
model done) by working on a clean and very readable code. In general,
the only drawback is a "structured" approach at the beginning which
certainly requires considerable design experience and time. Precisely
this, however, then allows an "agile" and very fast approach in tests,
in customer feedback, in reworks. I don't think DDD (neither
microservices, nor ORMs, nor noSql ...) is the silver bullet that solves
all problems. But we can intelligently take advantage of these
approaches without marrying them.
I think it would be very interesting if some community developer were
interested in creating a * small * POC of a DDD project. Using the OFBiz
Entity Engine by exploiting the DDD concepts and therefore the
generation of entity objects, value objects, repositories ... We could
take a "domain", then a small part of OFBiz data-model and experiment
with this approach. Leaving aside the graphical interface part, focusing
only on the core.
What do you think?
Regards,
Nicola
On 01/03/22 04:56, Aditya Sharma wrote:
> Hi Taher,
>
> Thanks for this initiative!
>
> How about incubating it and adopting it as a subproject?
>
> Thanks and Regards,
> Aditya Sharma
>
> On Tue, Jan 25, 2022 at 10:28 PM Eugen Stan<eu...@netdava.com> wrote:
>
>> Hello Taher,
>>
>> Thank you for this initiative and sorry for not replying sooner.
>> I meant to reply for so long but life ...
>>
>> I think most / all agrees that change is needed in OFBiz but I guess we
>> have different ideas on how to proceed.
>>
>> I don't think I can add a lot to what was already said.
>>
>> I do think keeping the current number of git repositories is good.
>> More will add overhead.
>> I do believe that ofbiz should provide stable extension points so people
>> can deploy their own plugins and functionality.
>>
>> I'm also working on some ideas around OFBiz - I believe I can
>> re-implement OFBiz entity engine on top of relational algebra using
>> Calcite.
>>
>> More on this soon on the mailing list.
>>
>> Looking forward to seeing a PoC from you.
>> I hope we do get to improve OFBiz and make it even more appealing.
>>
>>
>> On 25.01.2022 16:29, Taher Alkhateeb wrote:
>>> Thank you everyone for your kind feedback and sharing your thoughts on
>>> this initiative.
>>>
>>> I didn't get enough feedback or momentum to give me the impression that
>>> this can be a community initiative. I think I made the discussion too
>>> early before coming up with a PoC. So I will attempt something privately
>>> and come back perhaps when it is in a more ready state and see if it
>>> garners any momentum. IF people are interested in collaborating, then
>>> perhaps you can approach me and we can team up.
>>>
>>> Thank you again.
>> Regards,
>>
>> --
>> Eugen Stan
>>
>> +40770 941 271 /https://www.netdava.com
--
*Nicola Mazzoni* | /CEO/CTO/
Mobile: +39 347 990 5529
*Mp Style srl*
Via Meucci, 37 | 41019 Limidi di Soliera (MO)
p.i / c.f IT03679300362
Re: OFBiz re-architecture thoughts
Posted by Aditya Sharma <ad...@apache.org>.
Hi Taher,
Thanks for this initiative!
How about incubating it and adopting it as a subproject?
Thanks and Regards,
Aditya Sharma
On Tue, Jan 25, 2022 at 10:28 PM Eugen Stan <eu...@netdava.com> wrote:
> Hello Taher,
>
> Thank you for this initiative and sorry for not replying sooner.
> I meant to reply for so long but life ...
>
> I think most / all agrees that change is needed in OFBiz but I guess we
> have different ideas on how to proceed.
>
> I don't think I can add a lot to what was already said.
>
> I do think keeping the current number of git repositories is good.
> More will add overhead.
> I do believe that ofbiz should provide stable extension points so people
> can deploy their own plugins and functionality.
>
> I'm also working on some ideas around OFBiz - I believe I can
> re-implement OFBiz entity engine on top of relational algebra using
> Calcite.
>
> More on this soon on the mailing list.
>
> Looking forward to seeing a PoC from you.
> I hope we do get to improve OFBiz and make it even more appealing.
>
>
> On 25.01.2022 16:29, Taher Alkhateeb wrote:
> > Thank you everyone for your kind feedback and sharing your thoughts on
> > this initiative.
> >
> > I didn't get enough feedback or momentum to give me the impression that
> > this can be a community initiative. I think I made the discussion too
> > early before coming up with a PoC. So I will attempt something privately
> > and come back perhaps when it is in a more ready state and see if it
> > garners any momentum. IF people are interested in collaborating, then
> > perhaps you can approach me and we can team up.
> >
> > Thank you again.
>
> Regards,
>
> --
> Eugen Stan
>
> +40770 941 271 / https://www.netdava.com
Re: OFBiz re-architecture thoughts
Posted by Eugen Stan <eu...@netdava.com>.
Hello Taher,
Thank you for this initiative and sorry for not replying sooner.
I meant to reply for so long but life ...
I think most / all agrees that change is needed in OFBiz but I guess we
have different ideas on how to proceed.
I don't think I can add a lot to what was already said.
I do think keeping the current number of git repositories is good.
More will add overhead.
I do believe that ofbiz should provide stable extension points so people
can deploy their own plugins and functionality.
I'm also working on some ideas around OFBiz - I believe I can
re-implement OFBiz entity engine on top of relational algebra using Calcite.
More on this soon on the mailing list.
Looking forward to seeing a PoC from you.
I hope we do get to improve OFBiz and make it even more appealing.
On 25.01.2022 16:29, Taher Alkhateeb wrote:
> Thank you everyone for your kind feedback and sharing your thoughts on
> this initiative.
>
> I didn't get enough feedback or momentum to give me the impression that
> this can be a community initiative. I think I made the discussion too
> early before coming up with a PoC. So I will attempt something privately
> and come back perhaps when it is in a more ready state and see if it
> garners any momentum. IF people are interested in collaborating, then
> perhaps you can approach me and we can team up.
>
> Thank you again.
Regards,
--
Eugen Stan
+40770 941 271 / https://www.netdava.com
Re: OFBiz re-architecture thoughts
Posted by Taher Alkhateeb <ta...@pythys.com.INVALID>.
Thank you everyone for your kind feedback and sharing your thoughts on
this initiative.
I didn't get enough feedback or momentum to give me the impression that
this can be a community initiative. I think I made the discussion too
early before coming up with a PoC. So I will attempt something privately
and come back perhaps when it is in a more ready state and see if it
garners any momentum. IF people are interested in collaborating, then
perhaps you can approach me and we can team up.
Thank you again.
On 1/9/22 11:16, Jacques Le Roux wrote:
> Hi,
>
> If we want to improve OFBiz w/o changing too much the current OFBiz
> why not create an official OFBiz2 fork and work to this goal there?
>
> Disclaimer: I don't think of doing something like that, if people are
> ready for it, why not?
>
> Jacques
>
> Le 28/12/2021 à 18:18, Sakthivel Vellingiri a écrit :
>> Thanks Taher and Michael for detailed insights and Sathish for
>> sharing your
>> thoughts on upgrades.
>>
>> Just to introduce myself, we are using OFBiz for little over 7 years
>> for a
>> Life Sciences Product, and we consider ourselves as end user and haven't
>> inclined to make hardcore changes to the framework as yet.
>>
>> 1. Lack of Code Isolation_:::
>> Our approach has been as outlined by Michael, we have custom
>> plugins
>> for each application and in some cases custom application plugins for
>> each
>> customer which are overrides and extensions of the OOTB OFBiz
>> application
>> plugins and we more or less leave the framework untouched except for
>> a few
>> issues we have discovered, and we are guilty of not contributing the fix
>> back to OFBIZ and hence maintaining in our own branch, but without
>> taking
>> an inventory now, my recollection is that the changes to the
>> framework is
>> minimal. we merge the changes from our customer application plugin
>> branch
>> to *our* common application plugin branch that way *our* common
>> application
>> plugin branch at a given point in time has most of *our* custom
>> application
>> plugin changes.
>>
>> 2. Rest Interface:::
>> We are currently using HTML based web UI and relying on
>> widget/freemarker templating for the changes. We are inclined to client
>> rendering using Vue Js, and having Rest API integrated in the core
>> framework would be a huge plus. but we are open to using Rest API as a
>> plugin to begin with, I know there has been some communication around
>> Rest
>> based webservice api in the email thread, I have to admit, we haven't
>> explored that all that much yet. But if a Rest API plugin is made
>> available, we will be happy to utilize that and share feedback.
>>
>> 3. UI Screens:::
>> As experienced by new users and outlined in this thread, we end up
>> customizing the UI to minimize the field display to our user needs
>> either
>> by making changes in minilang or ftl, Perhaps it might be good idea
>> to have
>> database tables to control what is rendered in the UI and perhaps
>> rendering
>> based on database configuration can include support for role/permission
>> based rendering to show different fields for different
>> roles/SecurityPermission.
>>
>> 4. Webapps :::
>> As of right now, we switch off the webapps that we do not need
>> currently, At less frequent times we see application logouts when
>> switching
>> from one webapp to another, i'm not sure if this caused by having
>> different
>> plugins as different webapps and passing the session id between them
>> and if
>> this be solved by having single webapp, but i do not understand enough
>> about the merits of having a single webapp at this time to comment
>> more on
>> this.
>>
>> I do not have strong opinions about the other points other than what is
>> discussed already and i will skip responding to them,
>>
>> In my opinion replacing OFBiz framework code with established open
>> source
>> which is purpose-built for a specific use case like Apache Shiro for
>> Security, eHCache for caching could bring down OFBiz framework code and
>> thereby maintenance of OFBiz, framework however i also recognize and
>> reflect the thoughts outlined in this thread that we need to be very
>> careful about bringing such hardcore changes with careful planning &
>> quality control not to disrupt the existing implementations.
>>
>> Thanks to everyone sharing their view points and starting a
>> conversation in
>> lieu of the future of OFBiz.
>>
>> just my 2 cents from the end user perspective.
>>
>> regards
>> Sakthi
>>
>>
>> On Mon, Dec 27, 2021 at 1:09 PM Taher Alkhateeb
>> <ta...@pythys.com.invalid>
>> wrote:
>>
>>> Hi Michael,
>>>
>>> Thank you so much for spending your time reading and sifting through
>>> everything, I really appreciate and value your feedback.
>>>
>>> I need to highlight that we deployed mixed and somewhat crazy solutions
>>> including multiple mobile apps, live video / chat streaming systems and
>>> other projects that are booming in Kuwait and exploding both in user
>>> bases and technical requirements which is why we cannot operate some of
>>> these projects the way we used to in the past.
>>>
>>> Anyway I think I'm not going to elaborate too much as we wait for more
>>> opinions from the community to see if this is worth pushing. However, I
>>> want to reply to a few points you raised:
>>>
>>> - There is a huge difference between _few_ modifications, and ZERO
>>> modifications to the framework or core components. This way you don't
>>> even need to freeze a version of OFBiz to develop against at all!
>>> - Breaking to multiple git repositories means having something like
>>> ./gradlew gitPullAll to update and you're done. Nothing breaks, no
>>> surprises. You have to try it to believe it :) I have client projects
>>> that have 15 git repos and I can upgrade with my eyes closed. Also
>>> there
>>> is always entanglement that you don't feel or care for until you
>>> actually cut the repo into pieces and then realize we have too much
>>> entanglement. Having many small git repositories is in fact the most
>>> enjoyable part of how we're writing software now and I can assign
>>> different modules to different teams and they work independent of each
>>> other. It's just a totally different ballgame.
>>> - All or nothing in terms of code means you have to onboard a lot of
>>> code that you don't need. I'm not saying you cannot it turn it off, I'm
>>> saying it should be opt-in, not opt-out. All this code is potential
>>> maintenance, bugs, security, and so on that you shouldn't carry on if
>>> you don't want it. This makes things the messiest in terms of seed /
>>> demo data but also in other entanglements.
>>> - Yes entities and services are re-usable but you're limited to only
>>> that. Powerful re-usability is applied to everything (configurations,
>>> routers, widgets, scripts, etc ...). What is mostly re-usable in OFBiz
>>> is entities and services, the others are mostly "usable" not
>>> "reusable".
>>> It would take too long to explain and provide examples so I'll delay
>>> this part.
>>> - It's so awesome to use groovy exactly where you need it, without
>>> having to jump into yet another file to only write 2 lines of code. I
>>> have a service that needs 5-6 lines of code, why do I have to point to
>>> an external script. Put the implementation RIGHT THERE next to the
>>> definition of your service for example, the same is true everywhere you
>>> need XML actions. So scripting in lacking in XML actions, and also they
>>> are inconsistent (works differently in different places).
>>> - Mixing XML with groovy is just part of the flexibility and having
>>> your
>>> code where you need it. Even debugging XML code improves by putting
>>> groovy snippets above / below to investigate the context. Nobody would
>>> want to do that by opening a new file just to print something from the
>>> context for example.
>>> - Services are pure business logic, controller is routing logic.
>>> Routing
>>> logic does not mean NO logic, it means logic that is focused on
>>> coordination and decisions on where to go and what to do and putting
>>> that logic in services is actually mixing layers and making the code
>>> harder to understand / debug. I cannot show you how severely limited
>>> routing logic is in OFBiz is without showing you 5 / 6 examples of
>>> powerful things we've done that we simply can not do in OFBiz.
>>> - We should IMHO by default have ONLY one web-app with the ability to
>>> add more in an "opt-in" fashion. However, this means we need to be able
>>> to break the screen definitions and routes, routing needs a major
>>> change
>>> to achieve this objective. The controller.xml would simply not cut it.
>>> Again I need examples to really highlight what I mean and I might do it
>>> later.
>>> - It is not enough to integrate REST to be able to have dynamic
>>> libraries integrated with OFBiz if you want that to work with your
>>> widgets and other constructs. You also need multiple rendering modes
>>> that allow you to select from a variety of different technologies /
>>> options with different macros and different theming constructs. Again I
>>> cannot say more without examples to illustrate clearly.
>>> - Finally, many of the changes that I'm proposing require substantial
>>> changes to the framework and redesigning it. Well we cannot do that
>>> without first refactoring it, which means we have to get rid of /
>>> update
>>> lots of code. which means we need to start using libraries to control
>>> the large size of that thing. It's a chain, one thing pulling another
>>> thing, which makes it inevitable that a major shakeup of the core
>>> framework is necessary. I'm not proposing to change it just to change
>>> it. I'm proposing changing it because without that many of the things
>>> I'm proposing simply won't happen.
>>>
>>> I can understand why you might not agree with some of the points. In a
>>> way it's kind of hard to imagine a different way of doing things when
>>> we're used to working a certain way, but once we get exposed we realize
>>> that OH .. I've been torturing myself :). However, I do so your
>>> very-good point on being conservative because there is substantial risk
>>> in doing everything that I suggest.
>>>
>>> Anyway opening this to the rest of the community and thank you in
>>> advance for sharing your thoughts.
>>>
>>> Cheers,
>>>
>>> Taher Alkhateeb
>>>
>>> On 12/27/21 17:20, Michael Brohl wrote:
>>>> Hi Taher,
>>>>
>>>> thanks for bringing this up! I will try to briefly explain my views on
>>>> the raised topics.
>>>>
>>>> My perspective is from nearly 20 years of intensive work with OFBiz,
>>>> both for internal and external projects. Those projects have/had a
>>>> volume reaching from about 50 to over 1000 project days and we have
>>>> active customer projects which are running for almost 20 years now,
>>>> still being developed further with the business. OFBiz often became a
>>>> crucial part of the business ecosystem of those users. It is also the
>>>> central toolbox for our services around development of web based
>>>> software solutions and system integration in our own company since 7
>>>> years now. I consider myself both a user of OFBiz and a service
>>>> provider/integrator.
>>>>
>>>> After so many years of working with OFBiz if have learned that - from
>>>> a business users perspective - the functionality and integration
>>>> aspects along with the powerful datamodel, services etc. as well as
>>>> flexibility, performance and scalability is way more important than
>>>> detailed technical aspects (in the sense of “modern” or “newest
>>>> technology”).
>>>>
>>>> As a service provider for OFBiz, we are feeling responsible to provide
>>>> a reliable, constant and upgradeable software ecosystem which is why
>>>> we are on a more "conservative" path regarding big or long running
>>>> changes in the project. I have seen several new, ambititious
>>>> contributors who wanted to change everything, using the newest
>>>> frameworks and tools around etc., but often did not have an extensive
>>>> background of real world projects or who are not able to explain why
>>>> they want to use other tools and mechanisms and which user problems
>>>> they really want to solve.
>>>>
>>>> Most of these people are not active anymore or unfinished solutions
>>>> are put into the codebase which is another reason why I am really
>>>> careful when it comes to massive changes. I think the project needs
>>>> good planning, a good migration path and constant support when those
>>>> changes are decided to take place.
>>>>
>>>> All in all, I do not feel such a great pain as you seem to feel but
>>>> there is definetely much room for improvement, for sure.
>>>>
>>>> I will try to give my views on the different topics inline. They are
>>>> brief and need/deserve more in-depth preparation from my side which I
>>>> cannot spare at the moment.
>>>>
>>>> For further discussions I recommend to split those topics in different
>>>> threads because they would become unreadable soon with more
>>>> participants. Also writing up the outcome of the discussions and the
>>>> further roadmap in the wiki might help.
>>>>
>>>> Let's see how it works.
>>>>
>>>> Michael Brohl
>>>>
>>>> ecomify GmbH - www.ecomify.de
>>>>
>>>>
>>>> Am 22.12.21 um 18:23 schrieb Taher Alkhateeb:
>>>>> Problems
>>>> ...
>>>>> _1. Lack of Code Isolation_:It is unlikely to do anything substantial
>>>>> in OFBiz without touching the core repositories. Compare this with
>>>>> moqui or some other web frameworks like say Ruby-On-Rails or django
>>>>> and you will see that you can work independently of the development
>>>>> path in these frameworks. This makes upgrades quite painfulwith
>>>>> patches and manual interventions. Some in the community even
>>>>> introduced a whole patching framework in the past.
>>>> We do not have this experience, at least not as much as you seem to
>>>> have. Customer projects are developed completely in plugins, using the
>>>> framework and avoiding changes there. There ARE some changes tagged as
>>>> mods but not too much to be a real pain during a vendor update. Some
>>>> mods are also contributed back to OFBiz so these mods disappear during
>>>> the next vendor import.
>>>>
>>>> We are not directly using the original framework but rather use the
>>>> original as an upstream branch for our own projects. This makes it
>>>> easy to track changes and merge them into our codebase.
>>>>
>>>> It would be helpful to have some specific examples on where you are
>>>> having those painful changes.
>>>>
>>>>> _2. All or nothing:_In OFBiz, if you want to use the system, either
>>>>> take the whole thing or take nothing. What if I don't want any user
>>>>> interface for example, and just want to utilize the entities and
>>>>> services?
>>>> We are running projects which do not use the original user interface
>>>> or are being headless / just REST interfaces. You are not forced to
>>>> use the UI and you can switch off the web apps, do we really have a
>>>> problem there?
>>>>
>>>>> _3. Limited Re-usability:_Because of the way current components are
>>>>> designed, it's difficult to take advantage of components in other
>>>>> components. Sharing artifacts is pretty much limited to entities and
>>>>> services. Even then they're not very shareable because they're
>>>>> designed in a non-usable way for a certain context and certain
>>>>> settings and database parameters.*
>>>> Datamodel and services, the core business artifacts, are shareable and
>>>> in my view that's what's needed mostly. Can you be more specific
>>>> what's missing or designed in a way that it is not (re)usable ?
>>>>
>>>>
>>>>> *
>>>>>
>>>>> _4. Fragmented Actions:_**Instead of having unified XML actions, you
>>>>> have fragmentation across different contexts (screens, services,
>>>>> SECAs, etc). Furthermore, you cannot mix XML actions with groovy or
>>>>> other artifacts except by calling into external scripts and things
>>>>> like that. In comparison to moqui which is more flexible in XML
>>>>> actions I find it restrictive to work with OFBiz actions and it's
>>>>> hard to review, debug, or modify. I'm listing something below as an
>>>>> example for comparison [1]*
>>>> I will have to investigate more how Moqui does those things. I'm not
>>>> sure what the example shows. Is it part of the controller logic?
>>>>
>>>> Generally, I think that it is a strength to have these tasks separated
>>>> in an MVC pattern and not being able to mix controller logic with
>>>> business logic.
>>>>
>>>> When it comes to debugging: how do you debug the shown XML code?
>>>>
>>>>
>>>>> *
>>>>>
>>>>> _5. Limited Routing capability:_**Routing is very minimal
>>>>> (controller.xml). It cannot contain business logic and is limited
>>>>> either to chaining requests or rendering screens. Sophisticated
>>>>> routing logic is important for making rich systems that react to
>>>>> differing circumstances.
>>>>> **
>>>> See above: the controller part should NOT contain too much business
>>>> logic. If the business logic would be mixed with the controller logic,
>>>> how would you then make the business logic more shareable (see point 3
>>>> above)?
>>>>
>>>> Maybe I'm missing something and some examples would help me understand
>>>> the point.
>>>>
>>>>
>>>>> _6. Core framework difficult to change:_**OFBiz core was written
>>>>> a/long/time ago, and back then you didn't have many libraries for
>>>>> caching, transactions, DB pooling, Concurrency stuff, and others, so
>>>>> a lot was hand crafted, and we have a lots of interconnected API that
>>>>> is difficult to change, and most of these APIs depend on static util
>>>>> files that just kept piling and piling over the years. Where is the
>>>>> service engine implementation exactly? What about the entity engine?
>>>>> What about widgets? They are all over the place! Pieces for XML
>>>>> parsing, others for DOM models, and so on. The code is scattered and
>>>>> fixing it means going through a maze of many classes that are
>>>>> interconnected and all referencing each other. It's also hard to
>>>>> identify the signature or figure out the "big picture" since you have
>>>>> essentially minimal interfaces and no clear public API that you can
>>>>> refactor against. Pretty much the whole thing is public API**[2].
>>>>> **
>>>> That's a big field and I think we should break this into subtopics to
>>>> discuss. Can you be more specific, for me this is too broad to
>>>> discuss. What is the real problem, what would be a better solution,
>>>> ...?
>>>>
>>>> In general, a framework like OFBiz IS complex, it will never be easy
>>>> to read/understand. And of course, there is code which should be
>>>> refactored and better maintained (deprecation etc.). It would maybe be
>>>> a good start to identify those and do refactorings there first to
>>>> reduce complexity and pave the path for bigger refactorings?
>>>>
>>>> From my experience, e.g. the caching capabilities of OFBiz are a big
>>>> strength and to change those parts, we need some really good proof of
>>>> concepts to check if other solutions are really better. I've had
>>>> several bad experiences with, for example, object relational mapping
>>>> frameworks (performance, complexity) and I'm still a fan of the entity
>>>> engine.
>>>>
>>>>
>>>>> _7. Web-apps design problem__:_ A big problem (in my opinion) in the
>>>>> design is around having multiple web-apps, one for each component (or
>>>>> more). This interrupts the flow of the app and makes it difficult to
>>>>> come up with smooth solutions that work across the various screens
>>>>> and instead we need to pass a key between the components (and face
>>>>> other problems around session management). This restricts you in what
>>>>> you can offer as a user experience.*
>>>>> *
>>>> Do you have a solution in mind to solve this and also keep those
>>>> applications separated (see above)? Single Sign On?
>>>>
>>>>
>>>>> _8. Problematic UI design__:_ Another/big/problem (in my opinion) is
>>>>> having the UI show everything (reference implementation). I never
>>>>> understood the value of showing/everything/! If I want to understand
>>>>> the data model, I will just look at the data model. The purpose of
>>>>> the UI is not to show everything, but to implement specific useful
>>>>> use-cases. The reference implementation we have is not doing that,
>>>>> and not offering something very useful, instead it is showing
>>>>> everything! To improve this, we need to switch from data-model based
>>>>> interface (party, order, etc ...) to use-case based interface
>>>>> (project management, e-commerce, ERP, CRM, etc ...)*
>>>>> *
>>>> That's indeed a big issue and part of the discussions with every new
>>>> user who wants to utilize the existing web apps. On one hand it is
>>>> great to be able to show the potential of the ecosystem, on the other
>>>> hand it's too complex, especially for new users.
>>>>
>>>> IMO this is not so much a problem of the framework in general, which
>>>> does not prevent us from developing better UI and workflows.
>>>> Personally, I see OFBiz as an ERP framework which has to be customized
>>>> to match the target users's needs and not so much as an
>>>> ready-to-use ERP.
>>>>
>>>> But yes, this can and should be improved with additional web apps or
>>>> even ONE web app which tightly integrates all core applications into a
>>>> one-stop ERP solution which is more user-friendly than the base
>>>> applications. On the other hand, this will make separation more
>>>> difficult...
>>>>
>>>>
>>>>> _9. Limited client rendering:_Because it is difficult to refactor the
>>>>> framework, it's hard to pin-point where to go exactly to refactor all
>>>>> the code needed to support more client-rendering technology (react,
>>>>> vue, angular, etc ...) and make it interact smoothly with the
>>>>> back-end (drop-downs, complex forms, etc ...). A lot of work needs to
>>>>> happen both to the /themes and to /framework to allow and provide
>>>>> more flexibility (also moving things to DB). If this step is not done
>>>>> you will be stuck with your UI capabilities and won't catch up
>>>>> (unless you develop your own UI which makes the UI of the community
>>>>> redundant and you let go of the whole widget system anyway).*
>>>>> *
>>>> I guess it would help a lot if we have REST integrated into the core
>>>> framework and expose a proper REST API. For the HTML based web UI, I
>>>> have not much to change as the widget/freemarker templating gives us
>>>> much flexibility.
>>>>
>>>> What do you miss exactly / in which direction do you want to do
>>>> changes?
>>>>
>>>>
>>>>> _10. Security is difficult to implement:_One of my favorite features
>>>>> in moqui is having security baked in the artifacts themselves to
>>>>> avoid sprinkling all screens / services with security logic. I never
>>>>> even worry about writing security related stuff until way close to
>>>>> the end of delivering the project. So there needs to be a whole
>>>>> security layer that sits below all the artifacts such that we don't
>>>>> even need to write security code in the various screens and services
>>>>> and around certain entities.*
>>>>> *
>>>> Can you elaborate more on how you want to achieve fine grained
>>>> permissions for screens and services if not tied to the screen/service
>>>> configurations (having in mind that they should be reusable)?
>>>>
>>>>
>>>>> _11. Limited Integration Solutions__:_ I know we have now a REST
>>>>> component, but this is not the same as being a core feature of the
>>>>> framework (essential for any modern web framework). Why should it be
>>>>> tightly integrated? Many reasons: security, localization, mapping to
>>>>> service engine, mapping to entities, etc ...*
>>>>> *
>>>> I'm currently working on the rest-api plugin and I agree that it would
>>>> be great to have it in the framework instead of being a plugin.
>>>>
>>>>
>>>>> _12. No Specialization:_Because of the big repository, and having
>>>>> everything point to everything, it's hard to specialize. Now compare
>>>>> this to the approach we have in our work which is easier to
>>>>> specialize in. For example, I have a component for payment gateway
>>>>> integration, another one for twilio (SMS integration), and I have
>>>>> application components for each solution in its own repository (HR,
>>>>> insurance stuff, many others) and each is living in its own git repo
>>>>> yet they work together just fine and without touching one line of
>>>>> code in the framework or even core components (despite depending on
>>>>> them and using pieces of them). I can jump into new versions of the
>>>>> framework everyday without breaking a sweat. When I work on component
>>>>> X, I'm only thinking of that and nothing else. I don't have to worry
>>>>> about changing something that would crash another component or bring
>>>>> the system down (happened to me many times in the past).
>>>> For our projects, we were going the other way: we have framework +
>>>> plugins in the same repository instead of handling different
>>>> repositories. I think it would be a real pain to split the project in
>>>> dozens of different repositories only to bring them together to
>>>> integrate during development or deployment again.
>>>>
>>>> Can you describe how you are handling those different repositories in
>>>> an efficient way to get a better picture?
>>>>
>>>> As long as your work in a component/application/plugin and don't touch
>>>> the framework code I also do not see why you should break something
>>>> for other components? Of course, there ARE applications which are more
>>>> fundamental than other (e.g. party, security) and if those are changed
>>>> they might break other functionality. I see those applications
>>>> belonging to the framework more than being independent applications.
>>>>
>>>> I agree that we should try to identify false or unnecessary
>>>> dependencies though.
>>>>
>>>>
>>>>>
>>>>> Solutions
>>>>>
>>>>> Whatever solutions we come up with I think they're going to be
>>>>> difficult and would require community buy-in and support. I'm
>>>>> breaking this down into things to keep, update, add and remove as
>>>>> follows:
>>>>>
>>>>>
>>>>> Keep it
>>>>>
>>>>> - XSD signatures (so entity definitions, service definitions, XML
>>>>> actions though should expand)
>>>>> - Basic constructs of what is a component (with basic directory
>>>>> structure)
>>>>> - Data model
>>>>> - Existing services (core data-model services)
>>>>> - Java / groovy code that doesn't use core framework constructs or
>>>>> goes too deep into the internals.
>>>>> - Most used and useful APIs (Making queries, calling services, etc
>>>>> ...) although it should all be converted to interfaces with separate
>>>>> implementations where it's lacking.
>>>>>
>>>> Agree to all above.
>>>>
>>>>
>>>>> Improve It
>>>>>
>>>>> - Break the project to smaller git repositories.
>>>>> - framework
>>>>> - data model
>>>>> - services
>>>>> - common UI
>>>>> - use-case applications (ERP, CRM, e-commerce, etc ...)
>>>>> - plugins
>>>>> - entire framework directory needs refurbishment and big chunks
>>>>> probably require a delete / rewrite
>>>>> - the web architecture (get rid of web-apps)
>>>>> - the tests (lots of them need to be deleted anyway)
>>>>> - Routing needs to be programmatic with ability to inject code.
>>>>>
>>>>>
>>>>> Add It
>>>>>
>>>>> - Configuration override mechanism to remove the need to touch the
>>>>> core. Overrides include everything including routing, component
>>>>> registration, etc ...
>>>>> - REST integration into the core
>>>>> - Artifact based security that maps users / groups to artifacts
>>>>> without the need to inject security code into every screen
>>>>> - A super-set of XML actions that can be applied in all contexts and
>>>>> can embed groovy code
>>>>> - New, simple use-case based user interface. It needs to be designed
>>>>> such that it is easy to extend to avoid the all-or-nothing UI that
>>>>> currently exists.
>>>>>
>>>>>
>>>>> Remove It / Deprecate It / Overwrite It
>>>>>
>>>>> - Most of the existing UI (needs to be replaced with simpler use-case
>>>>> based interface)
>>>>> - Lots of the core framework code (needs major rewrites, needs
>>>>> interfaces, import libraries for things like transactions, security,
>>>>> concurrency, etc ...)
>>>>> - Probably some of the existing plugins might need to be deprecated.
>>>> See my questions above for some of the topics. I will add more details
>>>> and my point-of-view once we've split up the topics.
>>>>
>>>> The overall change would be huge and we should paint a clear picture
>>>> of what we want to achieve and which way to go in detail.
>>>>
>>>> We'll have to prioritize with a good cost/effort consideration and do
>>>> implementations step-by-step with a proper migration plan. Maybe
>>>> having feature branches for a longer time to be able to put out
>>>> releases if the development gets stuck or lasts longer.
>>>>
>>>>
>>>>>
>>>>> Quality Control
>>>>>
>>>>> Investing in such a huge effort requires that we change the way we
>>>>> view the project after / while work is being completed. Code should
>>>>> be approached with greater care and a lot of emphasis should be put
>>>>> on high quality, readability, avoiding anti-patterns, enforcing good
>>>>> patterns and best-practices. Some of the daily workflows need to
>>>>> improve / change, more thorough auditing and peer-reviews, and so on.
>>>> A big +1 for quality control. This will be a real challenge for the
>>>> project as it requires dedication and enough ressources for review.
>>>>
>>>>
>>>>>
>>>>> Tradeoffs
>>>>>
>>>>> It goes without saying that the above is a _lot_ of work and
>>>>> commitment and requires a lot of effort. That's why I hesitated in
>>>>> starting this thread, but I thought maybe showing the tradeoffs would
>>>>> make it easier to weigh things out and decide on a direction to push.
>>>>>
>>>>>
>>>>> Pros
>>>>>
>>>>> - Pretty much most of the above mentioned problems will be resolved.
>>>>> - The framework would be resilient and future proof.
>>>>> - We will have more options and capabilities in the future, it would
>>>>> be easier to incorporate new technologies as they emerge
>>>>> - Development would go faster
>>>>> - Technical Debt will be reduced
>>>>>
>>>>>
>>>>> Cons
>>>>>
>>>>> - Huge (maybe too huge) upfront costs in time and effort to make the
>>>>> switch
>>>>> - Extra effort needed to ensure a smooth upgrade path for existing
>>>>> users / adopters / system integrators. A whole layer of code needs to
>>>>> be written to ensure the ability to upgrade without massive rewrites.
>>>>> - Risk of failure, exhaustion, inability to push through to the end
>>>>> - Inconvenience to those with existing systems / solutions
>>>>>
>>>>>
>>>>> Summary
>>>>>
>>>>> Change is the only constant. Inability to adapt to the future means
>>>>> not surviving. I'm not confident what is the best change possible or
>>>>> at what pace, but I do know that lack of change is a death sentence
>>>>> to any project. Maybe it's time to reflect and think of where we'd
>>>>> like to go and this thread is sort of a brain-dump of possible ideas.
>>>>>
>>>>> I look forward to hearing your thoughts.
>>>> Thanks, Taher, for the thoughtful write-up of your view on the project
>>>> and the fields of change you see to be tackled. I do not agree with
>>>> all of them and I have a sensible view on cost/effort, thorough
>>>> planning and the need to always have existing users/projects in mind.
>>>> But I hope that we/the project will find a way to constantly work on
>>>> improvements.
>>>>
>>>> Again, I would propose to split up the most important topics into
>>>> threads with accompanying documentation in the wiki to help avoid
>>>> confusion.
>>>>
>>>> I am curious what others have to say and if we can activate enough
>>>> interest to improve.
>>>>
>>>>
>>>>> Cheers,
>>>>>
>>>>> Taher Alkhateeb
>>>>>
>>>>>
>>>>> [1] Example mixed XML actions and groovy from moqui
>>>>>
>>>>> <if condition="settlementTerm?.orderPmtServiceRegisterId">
>>>>> <then>
>>>>> <service-call name="update#mantle.order.OrderPart"
>>>>> in-map="[orderId:orderId,
>>>>> orderPartSeqId:orderPartSeqId, settlementTermId:settlementTermId]"/>
>>>>>
>>>>> <entity-find-one entity-name="moqui.service.ServiceRegister"
>>>>> value-field="serviceRegister">
>>>>> <field-map field-name="serviceRegisterId"
>>>>> from="settlementTerm.orderPmtServiceRegisterId"/>
>>>>> </entity-find-one>
>>>>> <script><![CDATA[
>>>>> try {
>>>>>
>>> ec.service.sync().name(serviceRegister.serviceName).parameters(context).softValidate(true).disableAuthz().call()
>>>
>>>
>>>>> if (ec.message.hasError()) return
>>>>> } catch (Throwable t) {
>>>>> ec.logger.log(300, "Error calling auto order
>>>>> payments
>>>>> service for SettlementTerm ${settlementTermId}", t)
>>>>> }
>>>>> ]]></script>
>>>>> </then>
>>>>> <else>
>>>>> <service-call
>>>>> name="mantle.account.PaymentServices.create#Payment"
>>>>> out-map="context"
>>>>> in-map="context + [amount:amount,
>>>>> amountUomId:amountUomId, fromPartyId:fromPartyId,
>>>>> toPartyId:toPartyId]"/>
>>>>> </else>
>>>>> </if>
>>>>>
>>>>> [2] https://ci.apache.org/projects/ofbiz/site/trunk/javadocs/
>>>>>
Re: OFBiz re-architecture thoughts
Posted by Jacques Le Roux <ja...@les7arts.com>.
Hi,
If we want to improve OFBiz w/o changing too much the current OFBiz why not create an official OFBiz2 fork and work to this goal there?
Disclaimer: I don't think of doing something like that, if people are ready for it, why not?
Jacques
Le 28/12/2021 à 18:18, Sakthivel Vellingiri a écrit :
> Thanks Taher and Michael for detailed insights and Sathish for sharing your
> thoughts on upgrades.
>
> Just to introduce myself, we are using OFBiz for little over 7 years for a
> Life Sciences Product, and we consider ourselves as end user and haven't
> inclined to make hardcore changes to the framework as yet.
>
> 1. Lack of Code Isolation_:::
> Our approach has been as outlined by Michael, we have custom plugins
> for each application and in some cases custom application plugins for each
> customer which are overrides and extensions of the OOTB OFBiz application
> plugins and we more or less leave the framework untouched except for a few
> issues we have discovered, and we are guilty of not contributing the fix
> back to OFBIZ and hence maintaining in our own branch, but without taking
> an inventory now, my recollection is that the changes to the framework is
> minimal. we merge the changes from our customer application plugin branch
> to *our* common application plugin branch that way *our* common application
> plugin branch at a given point in time has most of *our* custom application
> plugin changes.
>
> 2. Rest Interface:::
> We are currently using HTML based web UI and relying on
> widget/freemarker templating for the changes. We are inclined to client
> rendering using Vue Js, and having Rest API integrated in the core
> framework would be a huge plus. but we are open to using Rest API as a
> plugin to begin with, I know there has been some communication around Rest
> based webservice api in the email thread, I have to admit, we haven't
> explored that all that much yet. But if a Rest API plugin is made
> available, we will be happy to utilize that and share feedback.
>
> 3. UI Screens:::
> As experienced by new users and outlined in this thread, we end up
> customizing the UI to minimize the field display to our user needs either
> by making changes in minilang or ftl, Perhaps it might be good idea to have
> database tables to control what is rendered in the UI and perhaps rendering
> based on database configuration can include support for role/permission
> based rendering to show different fields for different
> roles/SecurityPermission.
>
> 4. Webapps :::
> As of right now, we switch off the webapps that we do not need
> currently, At less frequent times we see application logouts when switching
> from one webapp to another, i'm not sure if this caused by having different
> plugins as different webapps and passing the session id between them and if
> this be solved by having single webapp, but i do not understand enough
> about the merits of having a single webapp at this time to comment more on
> this.
>
> I do not have strong opinions about the other points other than what is
> discussed already and i will skip responding to them,
>
> In my opinion replacing OFBiz framework code with established open source
> which is purpose-built for a specific use case like Apache Shiro for
> Security, eHCache for caching could bring down OFBiz framework code and
> thereby maintenance of OFBiz, framework however i also recognize and
> reflect the thoughts outlined in this thread that we need to be very
> careful about bringing such hardcore changes with careful planning &
> quality control not to disrupt the existing implementations.
>
> Thanks to everyone sharing their view points and starting a conversation in
> lieu of the future of OFBiz.
>
> just my 2 cents from the end user perspective.
>
> regards
> Sakthi
>
>
> On Mon, Dec 27, 2021 at 1:09 PM Taher Alkhateeb <ta...@pythys.com.invalid>
> wrote:
>
>> Hi Michael,
>>
>> Thank you so much for spending your time reading and sifting through
>> everything, I really appreciate and value your feedback.
>>
>> I need to highlight that we deployed mixed and somewhat crazy solutions
>> including multiple mobile apps, live video / chat streaming systems and
>> other projects that are booming in Kuwait and exploding both in user
>> bases and technical requirements which is why we cannot operate some of
>> these projects the way we used to in the past.
>>
>> Anyway I think I'm not going to elaborate too much as we wait for more
>> opinions from the community to see if this is worth pushing. However, I
>> want to reply to a few points you raised:
>>
>> - There is a huge difference between _few_ modifications, and ZERO
>> modifications to the framework or core components. This way you don't
>> even need to freeze a version of OFBiz to develop against at all!
>> - Breaking to multiple git repositories means having something like
>> ./gradlew gitPullAll to update and you're done. Nothing breaks, no
>> surprises. You have to try it to believe it :) I have client projects
>> that have 15 git repos and I can upgrade with my eyes closed. Also there
>> is always entanglement that you don't feel or care for until you
>> actually cut the repo into pieces and then realize we have too much
>> entanglement. Having many small git repositories is in fact the most
>> enjoyable part of how we're writing software now and I can assign
>> different modules to different teams and they work independent of each
>> other. It's just a totally different ballgame.
>> - All or nothing in terms of code means you have to onboard a lot of
>> code that you don't need. I'm not saying you cannot it turn it off, I'm
>> saying it should be opt-in, not opt-out. All this code is potential
>> maintenance, bugs, security, and so on that you shouldn't carry on if
>> you don't want it. This makes things the messiest in terms of seed /
>> demo data but also in other entanglements.
>> - Yes entities and services are re-usable but you're limited to only
>> that. Powerful re-usability is applied to everything (configurations,
>> routers, widgets, scripts, etc ...). What is mostly re-usable in OFBiz
>> is entities and services, the others are mostly "usable" not "reusable".
>> It would take too long to explain and provide examples so I'll delay
>> this part.
>> - It's so awesome to use groovy exactly where you need it, without
>> having to jump into yet another file to only write 2 lines of code. I
>> have a service that needs 5-6 lines of code, why do I have to point to
>> an external script. Put the implementation RIGHT THERE next to the
>> definition of your service for example, the same is true everywhere you
>> need XML actions. So scripting in lacking in XML actions, and also they
>> are inconsistent (works differently in different places).
>> - Mixing XML with groovy is just part of the flexibility and having your
>> code where you need it. Even debugging XML code improves by putting
>> groovy snippets above / below to investigate the context. Nobody would
>> want to do that by opening a new file just to print something from the
>> context for example.
>> - Services are pure business logic, controller is routing logic. Routing
>> logic does not mean NO logic, it means logic that is focused on
>> coordination and decisions on where to go and what to do and putting
>> that logic in services is actually mixing layers and making the code
>> harder to understand / debug. I cannot show you how severely limited
>> routing logic is in OFBiz is without showing you 5 / 6 examples of
>> powerful things we've done that we simply can not do in OFBiz.
>> - We should IMHO by default have ONLY one web-app with the ability to
>> add more in an "opt-in" fashion. However, this means we need to be able
>> to break the screen definitions and routes, routing needs a major change
>> to achieve this objective. The controller.xml would simply not cut it.
>> Again I need examples to really highlight what I mean and I might do it
>> later.
>> - It is not enough to integrate REST to be able to have dynamic
>> libraries integrated with OFBiz if you want that to work with your
>> widgets and other constructs. You also need multiple rendering modes
>> that allow you to select from a variety of different technologies /
>> options with different macros and different theming constructs. Again I
>> cannot say more without examples to illustrate clearly.
>> - Finally, many of the changes that I'm proposing require substantial
>> changes to the framework and redesigning it. Well we cannot do that
>> without first refactoring it, which means we have to get rid of / update
>> lots of code. which means we need to start using libraries to control
>> the large size of that thing. It's a chain, one thing pulling another
>> thing, which makes it inevitable that a major shakeup of the core
>> framework is necessary. I'm not proposing to change it just to change
>> it. I'm proposing changing it because without that many of the things
>> I'm proposing simply won't happen.
>>
>> I can understand why you might not agree with some of the points. In a
>> way it's kind of hard to imagine a different way of doing things when
>> we're used to working a certain way, but once we get exposed we realize
>> that OH .. I've been torturing myself :). However, I do so your
>> very-good point on being conservative because there is substantial risk
>> in doing everything that I suggest.
>>
>> Anyway opening this to the rest of the community and thank you in
>> advance for sharing your thoughts.
>>
>> Cheers,
>>
>> Taher Alkhateeb
>>
>> On 12/27/21 17:20, Michael Brohl wrote:
>>> Hi Taher,
>>>
>>> thanks for bringing this up! I will try to briefly explain my views on
>>> the raised topics.
>>>
>>> My perspective is from nearly 20 years of intensive work with OFBiz,
>>> both for internal and external projects. Those projects have/had a
>>> volume reaching from about 50 to over 1000 project days and we have
>>> active customer projects which are running for almost 20 years now,
>>> still being developed further with the business. OFBiz often became a
>>> crucial part of the business ecosystem of those users. It is also the
>>> central toolbox for our services around development of web based
>>> software solutions and system integration in our own company since 7
>>> years now. I consider myself both a user of OFBiz and a service
>>> provider/integrator.
>>>
>>> After so many years of working with OFBiz if have learned that - from
>>> a business users perspective - the functionality and integration
>>> aspects along with the powerful datamodel, services etc. as well as
>>> flexibility, performance and scalability is way more important than
>>> detailed technical aspects (in the sense of “modern” or “newest
>>> technology”).
>>>
>>> As a service provider for OFBiz, we are feeling responsible to provide
>>> a reliable, constant and upgradeable software ecosystem which is why
>>> we are on a more "conservative" path regarding big or long running
>>> changes in the project. I have seen several new, ambititious
>>> contributors who wanted to change everything, using the newest
>>> frameworks and tools around etc., but often did not have an extensive
>>> background of real world projects or who are not able to explain why
>>> they want to use other tools and mechanisms and which user problems
>>> they really want to solve.
>>>
>>> Most of these people are not active anymore or unfinished solutions
>>> are put into the codebase which is another reason why I am really
>>> careful when it comes to massive changes. I think the project needs
>>> good planning, a good migration path and constant support when those
>>> changes are decided to take place.
>>>
>>> All in all, I do not feel such a great pain as you seem to feel but
>>> there is definetely much room for improvement, for sure.
>>>
>>> I will try to give my views on the different topics inline. They are
>>> brief and need/deserve more in-depth preparation from my side which I
>>> cannot spare at the moment.
>>>
>>> For further discussions I recommend to split those topics in different
>>> threads because they would become unreadable soon with more
>>> participants. Also writing up the outcome of the discussions and the
>>> further roadmap in the wiki might help.
>>>
>>> Let's see how it works.
>>>
>>> Michael Brohl
>>>
>>> ecomify GmbH - www.ecomify.de
>>>
>>>
>>> Am 22.12.21 um 18:23 schrieb Taher Alkhateeb:
>>>> Problems
>>> ...
>>>> _1. Lack of Code Isolation_:It is unlikely to do anything substantial
>>>> in OFBiz without touching the core repositories. Compare this with
>>>> moqui or some other web frameworks like say Ruby-On-Rails or django
>>>> and you will see that you can work independently of the development
>>>> path in these frameworks. This makes upgrades quite painfulwith
>>>> patches and manual interventions. Some in the community even
>>>> introduced a whole patching framework in the past.
>>> We do not have this experience, at least not as much as you seem to
>>> have. Customer projects are developed completely in plugins, using the
>>> framework and avoiding changes there. There ARE some changes tagged as
>>> mods but not too much to be a real pain during a vendor update. Some
>>> mods are also contributed back to OFBiz so these mods disappear during
>>> the next vendor import.
>>>
>>> We are not directly using the original framework but rather use the
>>> original as an upstream branch for our own projects. This makes it
>>> easy to track changes and merge them into our codebase.
>>>
>>> It would be helpful to have some specific examples on where you are
>>> having those painful changes.
>>>
>>>> _2. All or nothing:_In OFBiz, if you want to use the system, either
>>>> take the whole thing or take nothing. What if I don't want any user
>>>> interface for example, and just want to utilize the entities and
>>>> services?
>>> We are running projects which do not use the original user interface
>>> or are being headless / just REST interfaces. You are not forced to
>>> use the UI and you can switch off the web apps, do we really have a
>>> problem there?
>>>
>>>> _3. Limited Re-usability:_Because of the way current components are
>>>> designed, it's difficult to take advantage of components in other
>>>> components. Sharing artifacts is pretty much limited to entities and
>>>> services. Even then they're not very shareable because they're
>>>> designed in a non-usable way for a certain context and certain
>>>> settings and database parameters.*
>>> Datamodel and services, the core business artifacts, are shareable and
>>> in my view that's what's needed mostly. Can you be more specific
>>> what's missing or designed in a way that it is not (re)usable ?
>>>
>>>
>>>> *
>>>>
>>>> _4. Fragmented Actions:_**Instead of having unified XML actions, you
>>>> have fragmentation across different contexts (screens, services,
>>>> SECAs, etc). Furthermore, you cannot mix XML actions with groovy or
>>>> other artifacts except by calling into external scripts and things
>>>> like that. In comparison to moqui which is more flexible in XML
>>>> actions I find it restrictive to work with OFBiz actions and it's
>>>> hard to review, debug, or modify. I'm listing something below as an
>>>> example for comparison [1]*
>>> I will have to investigate more how Moqui does those things. I'm not
>>> sure what the example shows. Is it part of the controller logic?
>>>
>>> Generally, I think that it is a strength to have these tasks separated
>>> in an MVC pattern and not being able to mix controller logic with
>>> business logic.
>>>
>>> When it comes to debugging: how do you debug the shown XML code?
>>>
>>>
>>>> *
>>>>
>>>> _5. Limited Routing capability:_**Routing is very minimal
>>>> (controller.xml). It cannot contain business logic and is limited
>>>> either to chaining requests or rendering screens. Sophisticated
>>>> routing logic is important for making rich systems that react to
>>>> differing circumstances.
>>>> **
>>> See above: the controller part should NOT contain too much business
>>> logic. If the business logic would be mixed with the controller logic,
>>> how would you then make the business logic more shareable (see point 3
>>> above)?
>>>
>>> Maybe I'm missing something and some examples would help me understand
>>> the point.
>>>
>>>
>>>> _6. Core framework difficult to change:_**OFBiz core was written
>>>> a/long/time ago, and back then you didn't have many libraries for
>>>> caching, transactions, DB pooling, Concurrency stuff, and others, so
>>>> a lot was hand crafted, and we have a lots of interconnected API that
>>>> is difficult to change, and most of these APIs depend on static util
>>>> files that just kept piling and piling over the years. Where is the
>>>> service engine implementation exactly? What about the entity engine?
>>>> What about widgets? They are all over the place! Pieces for XML
>>>> parsing, others for DOM models, and so on. The code is scattered and
>>>> fixing it means going through a maze of many classes that are
>>>> interconnected and all referencing each other. It's also hard to
>>>> identify the signature or figure out the "big picture" since you have
>>>> essentially minimal interfaces and no clear public API that you can
>>>> refactor against. Pretty much the whole thing is public API**[2].
>>>> **
>>> That's a big field and I think we should break this into subtopics to
>>> discuss. Can you be more specific, for me this is too broad to
>>> discuss. What is the real problem, what would be a better solution, ...?
>>>
>>> In general, a framework like OFBiz IS complex, it will never be easy
>>> to read/understand. And of course, there is code which should be
>>> refactored and better maintained (deprecation etc.). It would maybe be
>>> a good start to identify those and do refactorings there first to
>>> reduce complexity and pave the path for bigger refactorings?
>>>
>>> From my experience, e.g. the caching capabilities of OFBiz are a big
>>> strength and to change those parts, we need some really good proof of
>>> concepts to check if other solutions are really better. I've had
>>> several bad experiences with, for example, object relational mapping
>>> frameworks (performance, complexity) and I'm still a fan of the entity
>>> engine.
>>>
>>>
>>>> _7. Web-apps design problem__:_ A big problem (in my opinion) in the
>>>> design is around having multiple web-apps, one for each component (or
>>>> more). This interrupts the flow of the app and makes it difficult to
>>>> come up with smooth solutions that work across the various screens
>>>> and instead we need to pass a key between the components (and face
>>>> other problems around session management). This restricts you in what
>>>> you can offer as a user experience.*
>>>> *
>>> Do you have a solution in mind to solve this and also keep those
>>> applications separated (see above)? Single Sign On?
>>>
>>>
>>>> _8. Problematic UI design__:_ Another/big/problem (in my opinion) is
>>>> having the UI show everything (reference implementation). I never
>>>> understood the value of showing/everything/! If I want to understand
>>>> the data model, I will just look at the data model. The purpose of
>>>> the UI is not to show everything, but to implement specific useful
>>>> use-cases. The reference implementation we have is not doing that,
>>>> and not offering something very useful, instead it is showing
>>>> everything! To improve this, we need to switch from data-model based
>>>> interface (party, order, etc ...) to use-case based interface
>>>> (project management, e-commerce, ERP, CRM, etc ...)*
>>>> *
>>> That's indeed a big issue and part of the discussions with every new
>>> user who wants to utilize the existing web apps. On one hand it is
>>> great to be able to show the potential of the ecosystem, on the other
>>> hand it's too complex, especially for new users.
>>>
>>> IMO this is not so much a problem of the framework in general, which
>>> does not prevent us from developing better UI and workflows.
>>> Personally, I see OFBiz as an ERP framework which has to be customized
>>> to match the target users's needs and not so much as an ready-to-use ERP.
>>>
>>> But yes, this can and should be improved with additional web apps or
>>> even ONE web app which tightly integrates all core applications into a
>>> one-stop ERP solution which is more user-friendly than the base
>>> applications. On the other hand, this will make separation more
>>> difficult...
>>>
>>>
>>>> _9. Limited client rendering:_Because it is difficult to refactor the
>>>> framework, it's hard to pin-point where to go exactly to refactor all
>>>> the code needed to support more client-rendering technology (react,
>>>> vue, angular, etc ...) and make it interact smoothly with the
>>>> back-end (drop-downs, complex forms, etc ...). A lot of work needs to
>>>> happen both to the /themes and to /framework to allow and provide
>>>> more flexibility (also moving things to DB). If this step is not done
>>>> you will be stuck with your UI capabilities and won't catch up
>>>> (unless you develop your own UI which makes the UI of the community
>>>> redundant and you let go of the whole widget system anyway).*
>>>> *
>>> I guess it would help a lot if we have REST integrated into the core
>>> framework and expose a proper REST API. For the HTML based web UI, I
>>> have not much to change as the widget/freemarker templating gives us
>>> much flexibility.
>>>
>>> What do you miss exactly / in which direction do you want to do changes?
>>>
>>>
>>>> _10. Security is difficult to implement:_One of my favorite features
>>>> in moqui is having security baked in the artifacts themselves to
>>>> avoid sprinkling all screens / services with security logic. I never
>>>> even worry about writing security related stuff until way close to
>>>> the end of delivering the project. So there needs to be a whole
>>>> security layer that sits below all the artifacts such that we don't
>>>> even need to write security code in the various screens and services
>>>> and around certain entities.*
>>>> *
>>> Can you elaborate more on how you want to achieve fine grained
>>> permissions for screens and services if not tied to the screen/service
>>> configurations (having in mind that they should be reusable)?
>>>
>>>
>>>> _11. Limited Integration Solutions__:_ I know we have now a REST
>>>> component, but this is not the same as being a core feature of the
>>>> framework (essential for any modern web framework). Why should it be
>>>> tightly integrated? Many reasons: security, localization, mapping to
>>>> service engine, mapping to entities, etc ...*
>>>> *
>>> I'm currently working on the rest-api plugin and I agree that it would
>>> be great to have it in the framework instead of being a plugin.
>>>
>>>
>>>> _12. No Specialization:_Because of the big repository, and having
>>>> everything point to everything, it's hard to specialize. Now compare
>>>> this to the approach we have in our work which is easier to
>>>> specialize in. For example, I have a component for payment gateway
>>>> integration, another one for twilio (SMS integration), and I have
>>>> application components for each solution in its own repository (HR,
>>>> insurance stuff, many others) and each is living in its own git repo
>>>> yet they work together just fine and without touching one line of
>>>> code in the framework or even core components (despite depending on
>>>> them and using pieces of them). I can jump into new versions of the
>>>> framework everyday without breaking a sweat. When I work on component
>>>> X, I'm only thinking of that and nothing else. I don't have to worry
>>>> about changing something that would crash another component or bring
>>>> the system down (happened to me many times in the past).
>>> For our projects, we were going the other way: we have framework +
>>> plugins in the same repository instead of handling different
>>> repositories. I think it would be a real pain to split the project in
>>> dozens of different repositories only to bring them together to
>>> integrate during development or deployment again.
>>>
>>> Can you describe how you are handling those different repositories in
>>> an efficient way to get a better picture?
>>>
>>> As long as your work in a component/application/plugin and don't touch
>>> the framework code I also do not see why you should break something
>>> for other components? Of course, there ARE applications which are more
>>> fundamental than other (e.g. party, security) and if those are changed
>>> they might break other functionality. I see those applications
>>> belonging to the framework more than being independent applications.
>>>
>>> I agree that we should try to identify false or unnecessary
>>> dependencies though.
>>>
>>>
>>>>
>>>> Solutions
>>>>
>>>> Whatever solutions we come up with I think they're going to be
>>>> difficult and would require community buy-in and support. I'm
>>>> breaking this down into things to keep, update, add and remove as
>>>> follows:
>>>>
>>>>
>>>> Keep it
>>>>
>>>> - XSD signatures (so entity definitions, service definitions, XML
>>>> actions though should expand)
>>>> - Basic constructs of what is a component (with basic directory
>>>> structure)
>>>> - Data model
>>>> - Existing services (core data-model services)
>>>> - Java / groovy code that doesn't use core framework constructs or
>>>> goes too deep into the internals.
>>>> - Most used and useful APIs (Making queries, calling services, etc
>>>> ...) although it should all be converted to interfaces with separate
>>>> implementations where it's lacking.
>>>>
>>> Agree to all above.
>>>
>>>
>>>> Improve It
>>>>
>>>> - Break the project to smaller git repositories.
>>>> - framework
>>>> - data model
>>>> - services
>>>> - common UI
>>>> - use-case applications (ERP, CRM, e-commerce, etc ...)
>>>> - plugins
>>>> - entire framework directory needs refurbishment and big chunks
>>>> probably require a delete / rewrite
>>>> - the web architecture (get rid of web-apps)
>>>> - the tests (lots of them need to be deleted anyway)
>>>> - Routing needs to be programmatic with ability to inject code.
>>>>
>>>>
>>>> Add It
>>>>
>>>> - Configuration override mechanism to remove the need to touch the
>>>> core. Overrides include everything including routing, component
>>>> registration, etc ...
>>>> - REST integration into the core
>>>> - Artifact based security that maps users / groups to artifacts
>>>> without the need to inject security code into every screen
>>>> - A super-set of XML actions that can be applied in all contexts and
>>>> can embed groovy code
>>>> - New, simple use-case based user interface. It needs to be designed
>>>> such that it is easy to extend to avoid the all-or-nothing UI that
>>>> currently exists.
>>>>
>>>>
>>>> Remove It / Deprecate It / Overwrite It
>>>>
>>>> - Most of the existing UI (needs to be replaced with simpler use-case
>>>> based interface)
>>>> - Lots of the core framework code (needs major rewrites, needs
>>>> interfaces, import libraries for things like transactions, security,
>>>> concurrency, etc ...)
>>>> - Probably some of the existing plugins might need to be deprecated.
>>> See my questions above for some of the topics. I will add more details
>>> and my point-of-view once we've split up the topics.
>>>
>>> The overall change would be huge and we should paint a clear picture
>>> of what we want to achieve and which way to go in detail.
>>>
>>> We'll have to prioritize with a good cost/effort consideration and do
>>> implementations step-by-step with a proper migration plan. Maybe
>>> having feature branches for a longer time to be able to put out
>>> releases if the development gets stuck or lasts longer.
>>>
>>>
>>>>
>>>> Quality Control
>>>>
>>>> Investing in such a huge effort requires that we change the way we
>>>> view the project after / while work is being completed. Code should
>>>> be approached with greater care and a lot of emphasis should be put
>>>> on high quality, readability, avoiding anti-patterns, enforcing good
>>>> patterns and best-practices. Some of the daily workflows need to
>>>> improve / change, more thorough auditing and peer-reviews, and so on.
>>> A big +1 for quality control. This will be a real challenge for the
>>> project as it requires dedication and enough ressources for review.
>>>
>>>
>>>>
>>>> Tradeoffs
>>>>
>>>> It goes without saying that the above is a _lot_ of work and
>>>> commitment and requires a lot of effort. That's why I hesitated in
>>>> starting this thread, but I thought maybe showing the tradeoffs would
>>>> make it easier to weigh things out and decide on a direction to push.
>>>>
>>>>
>>>> Pros
>>>>
>>>> - Pretty much most of the above mentioned problems will be resolved.
>>>> - The framework would be resilient and future proof.
>>>> - We will have more options and capabilities in the future, it would
>>>> be easier to incorporate new technologies as they emerge
>>>> - Development would go faster
>>>> - Technical Debt will be reduced
>>>>
>>>>
>>>> Cons
>>>>
>>>> - Huge (maybe too huge) upfront costs in time and effort to make the
>>>> switch
>>>> - Extra effort needed to ensure a smooth upgrade path for existing
>>>> users / adopters / system integrators. A whole layer of code needs to
>>>> be written to ensure the ability to upgrade without massive rewrites.
>>>> - Risk of failure, exhaustion, inability to push through to the end
>>>> - Inconvenience to those with existing systems / solutions
>>>>
>>>>
>>>> Summary
>>>>
>>>> Change is the only constant. Inability to adapt to the future means
>>>> not surviving. I'm not confident what is the best change possible or
>>>> at what pace, but I do know that lack of change is a death sentence
>>>> to any project. Maybe it's time to reflect and think of where we'd
>>>> like to go and this thread is sort of a brain-dump of possible ideas.
>>>>
>>>> I look forward to hearing your thoughts.
>>> Thanks, Taher, for the thoughtful write-up of your view on the project
>>> and the fields of change you see to be tackled. I do not agree with
>>> all of them and I have a sensible view on cost/effort, thorough
>>> planning and the need to always have existing users/projects in mind.
>>> But I hope that we/the project will find a way to constantly work on
>>> improvements.
>>>
>>> Again, I would propose to split up the most important topics into
>>> threads with accompanying documentation in the wiki to help avoid
>>> confusion.
>>>
>>> I am curious what others have to say and if we can activate enough
>>> interest to improve.
>>>
>>>
>>>> Cheers,
>>>>
>>>> Taher Alkhateeb
>>>>
>>>>
>>>> [1] Example mixed XML actions and groovy from moqui
>>>>
>>>> <if condition="settlementTerm?.orderPmtServiceRegisterId">
>>>> <then>
>>>> <service-call name="update#mantle.order.OrderPart"
>>>> in-map="[orderId:orderId,
>>>> orderPartSeqId:orderPartSeqId, settlementTermId:settlementTermId]"/>
>>>>
>>>> <entity-find-one entity-name="moqui.service.ServiceRegister"
>>>> value-field="serviceRegister">
>>>> <field-map field-name="serviceRegisterId"
>>>> from="settlementTerm.orderPmtServiceRegisterId"/>
>>>> </entity-find-one>
>>>> <script><![CDATA[
>>>> try {
>>>>
>> ec.service.sync().name(serviceRegister.serviceName).parameters(context).softValidate(true).disableAuthz().call()
>>
>>>> if (ec.message.hasError()) return
>>>> } catch (Throwable t) {
>>>> ec.logger.log(300, "Error calling auto order payments
>>>> service for SettlementTerm ${settlementTermId}", t)
>>>> }
>>>> ]]></script>
>>>> </then>
>>>> <else>
>>>> <service-call
>>>> name="mantle.account.PaymentServices.create#Payment" out-map="context"
>>>> in-map="context + [amount:amount,
>>>> amountUomId:amountUomId, fromPartyId:fromPartyId,
>>>> toPartyId:toPartyId]"/>
>>>> </else>
>>>> </if>
>>>>
>>>> [2] https://ci.apache.org/projects/ofbiz/site/trunk/javadocs/
>>>>
Re: OFBiz re-architecture thoughts
Posted by Sakthivel Vellingiri <in...@gmail.com>.
Thanks Taher and Michael for detailed insights and Sathish for sharing your
thoughts on upgrades.
Just to introduce myself, we are using OFBiz for little over 7 years for a
Life Sciences Product, and we consider ourselves as end user and haven't
inclined to make hardcore changes to the framework as yet.
1. Lack of Code Isolation_:::
Our approach has been as outlined by Michael, we have custom plugins
for each application and in some cases custom application plugins for each
customer which are overrides and extensions of the OOTB OFBiz application
plugins and we more or less leave the framework untouched except for a few
issues we have discovered, and we are guilty of not contributing the fix
back to OFBIZ and hence maintaining in our own branch, but without taking
an inventory now, my recollection is that the changes to the framework is
minimal. we merge the changes from our customer application plugin branch
to *our* common application plugin branch that way *our* common application
plugin branch at a given point in time has most of *our* custom application
plugin changes.
2. Rest Interface:::
We are currently using HTML based web UI and relying on
widget/freemarker templating for the changes. We are inclined to client
rendering using Vue Js, and having Rest API integrated in the core
framework would be a huge plus. but we are open to using Rest API as a
plugin to begin with, I know there has been some communication around Rest
based webservice api in the email thread, I have to admit, we haven't
explored that all that much yet. But if a Rest API plugin is made
available, we will be happy to utilize that and share feedback.
3. UI Screens:::
As experienced by new users and outlined in this thread, we end up
customizing the UI to minimize the field display to our user needs either
by making changes in minilang or ftl, Perhaps it might be good idea to have
database tables to control what is rendered in the UI and perhaps rendering
based on database configuration can include support for role/permission
based rendering to show different fields for different
roles/SecurityPermission.
4. Webapps :::
As of right now, we switch off the webapps that we do not need
currently, At less frequent times we see application logouts when switching
from one webapp to another, i'm not sure if this caused by having different
plugins as different webapps and passing the session id between them and if
this be solved by having single webapp, but i do not understand enough
about the merits of having a single webapp at this time to comment more on
this.
I do not have strong opinions about the other points other than what is
discussed already and i will skip responding to them,
In my opinion replacing OFBiz framework code with established open source
which is purpose-built for a specific use case like Apache Shiro for
Security, eHCache for caching could bring down OFBiz framework code and
thereby maintenance of OFBiz, framework however i also recognize and
reflect the thoughts outlined in this thread that we need to be very
careful about bringing such hardcore changes with careful planning &
quality control not to disrupt the existing implementations.
Thanks to everyone sharing their view points and starting a conversation in
lieu of the future of OFBiz.
just my 2 cents from the end user perspective.
regards
Sakthi
On Mon, Dec 27, 2021 at 1:09 PM Taher Alkhateeb <ta...@pythys.com.invalid>
wrote:
> Hi Michael,
>
> Thank you so much for spending your time reading and sifting through
> everything, I really appreciate and value your feedback.
>
> I need to highlight that we deployed mixed and somewhat crazy solutions
> including multiple mobile apps, live video / chat streaming systems and
> other projects that are booming in Kuwait and exploding both in user
> bases and technical requirements which is why we cannot operate some of
> these projects the way we used to in the past.
>
> Anyway I think I'm not going to elaborate too much as we wait for more
> opinions from the community to see if this is worth pushing. However, I
> want to reply to a few points you raised:
>
> - There is a huge difference between _few_ modifications, and ZERO
> modifications to the framework or core components. This way you don't
> even need to freeze a version of OFBiz to develop against at all!
> - Breaking to multiple git repositories means having something like
> ./gradlew gitPullAll to update and you're done. Nothing breaks, no
> surprises. You have to try it to believe it :) I have client projects
> that have 15 git repos and I can upgrade with my eyes closed. Also there
> is always entanglement that you don't feel or care for until you
> actually cut the repo into pieces and then realize we have too much
> entanglement. Having many small git repositories is in fact the most
> enjoyable part of how we're writing software now and I can assign
> different modules to different teams and they work independent of each
> other. It's just a totally different ballgame.
> - All or nothing in terms of code means you have to onboard a lot of
> code that you don't need. I'm not saying you cannot it turn it off, I'm
> saying it should be opt-in, not opt-out. All this code is potential
> maintenance, bugs, security, and so on that you shouldn't carry on if
> you don't want it. This makes things the messiest in terms of seed /
> demo data but also in other entanglements.
> - Yes entities and services are re-usable but you're limited to only
> that. Powerful re-usability is applied to everything (configurations,
> routers, widgets, scripts, etc ...). What is mostly re-usable in OFBiz
> is entities and services, the others are mostly "usable" not "reusable".
> It would take too long to explain and provide examples so I'll delay
> this part.
> - It's so awesome to use groovy exactly where you need it, without
> having to jump into yet another file to only write 2 lines of code. I
> have a service that needs 5-6 lines of code, why do I have to point to
> an external script. Put the implementation RIGHT THERE next to the
> definition of your service for example, the same is true everywhere you
> need XML actions. So scripting in lacking in XML actions, and also they
> are inconsistent (works differently in different places).
> - Mixing XML with groovy is just part of the flexibility and having your
> code where you need it. Even debugging XML code improves by putting
> groovy snippets above / below to investigate the context. Nobody would
> want to do that by opening a new file just to print something from the
> context for example.
> - Services are pure business logic, controller is routing logic. Routing
> logic does not mean NO logic, it means logic that is focused on
> coordination and decisions on where to go and what to do and putting
> that logic in services is actually mixing layers and making the code
> harder to understand / debug. I cannot show you how severely limited
> routing logic is in OFBiz is without showing you 5 / 6 examples of
> powerful things we've done that we simply can not do in OFBiz.
> - We should IMHO by default have ONLY one web-app with the ability to
> add more in an "opt-in" fashion. However, this means we need to be able
> to break the screen definitions and routes, routing needs a major change
> to achieve this objective. The controller.xml would simply not cut it.
> Again I need examples to really highlight what I mean and I might do it
> later.
> - It is not enough to integrate REST to be able to have dynamic
> libraries integrated with OFBiz if you want that to work with your
> widgets and other constructs. You also need multiple rendering modes
> that allow you to select from a variety of different technologies /
> options with different macros and different theming constructs. Again I
> cannot say more without examples to illustrate clearly.
> - Finally, many of the changes that I'm proposing require substantial
> changes to the framework and redesigning it. Well we cannot do that
> without first refactoring it, which means we have to get rid of / update
> lots of code. which means we need to start using libraries to control
> the large size of that thing. It's a chain, one thing pulling another
> thing, which makes it inevitable that a major shakeup of the core
> framework is necessary. I'm not proposing to change it just to change
> it. I'm proposing changing it because without that many of the things
> I'm proposing simply won't happen.
>
> I can understand why you might not agree with some of the points. In a
> way it's kind of hard to imagine a different way of doing things when
> we're used to working a certain way, but once we get exposed we realize
> that OH .. I've been torturing myself :). However, I do so your
> very-good point on being conservative because there is substantial risk
> in doing everything that I suggest.
>
> Anyway opening this to the rest of the community and thank you in
> advance for sharing your thoughts.
>
> Cheers,
>
> Taher Alkhateeb
>
> On 12/27/21 17:20, Michael Brohl wrote:
> > Hi Taher,
> >
> > thanks for bringing this up! I will try to briefly explain my views on
> > the raised topics.
> >
> > My perspective is from nearly 20 years of intensive work with OFBiz,
> > both for internal and external projects. Those projects have/had a
> > volume reaching from about 50 to over 1000 project days and we have
> > active customer projects which are running for almost 20 years now,
> > still being developed further with the business. OFBiz often became a
> > crucial part of the business ecosystem of those users. It is also the
> > central toolbox for our services around development of web based
> > software solutions and system integration in our own company since 7
> > years now. I consider myself both a user of OFBiz and a service
> > provider/integrator.
> >
> > After so many years of working with OFBiz if have learned that - from
> > a business users perspective - the functionality and integration
> > aspects along with the powerful datamodel, services etc. as well as
> > flexibility, performance and scalability is way more important than
> > detailed technical aspects (in the sense of “modern” or “newest
> > technology”).
> >
> > As a service provider for OFBiz, we are feeling responsible to provide
> > a reliable, constant and upgradeable software ecosystem which is why
> > we are on a more "conservative" path regarding big or long running
> > changes in the project. I have seen several new, ambititious
> > contributors who wanted to change everything, using the newest
> > frameworks and tools around etc., but often did not have an extensive
> > background of real world projects or who are not able to explain why
> > they want to use other tools and mechanisms and which user problems
> > they really want to solve.
> >
> > Most of these people are not active anymore or unfinished solutions
> > are put into the codebase which is another reason why I am really
> > careful when it comes to massive changes. I think the project needs
> > good planning, a good migration path and constant support when those
> > changes are decided to take place.
> >
> > All in all, I do not feel such a great pain as you seem to feel but
> > there is definetely much room for improvement, for sure.
> >
> > I will try to give my views on the different topics inline. They are
> > brief and need/deserve more in-depth preparation from my side which I
> > cannot spare at the moment.
> >
> > For further discussions I recommend to split those topics in different
> > threads because they would become unreadable soon with more
> > participants. Also writing up the outcome of the discussions and the
> > further roadmap in the wiki might help.
> >
> > Let's see how it works.
> >
> > Michael Brohl
> >
> > ecomify GmbH - www.ecomify.de
> >
> >
> > Am 22.12.21 um 18:23 schrieb Taher Alkhateeb:
> >> Problems
> > ...
> >> _1. Lack of Code Isolation_:It is unlikely to do anything substantial
> >> in OFBiz without touching the core repositories. Compare this with
> >> moqui or some other web frameworks like say Ruby-On-Rails or django
> >> and you will see that you can work independently of the development
> >> path in these frameworks. This makes upgrades quite painfulwith
> >> patches and manual interventions. Some in the community even
> >> introduced a whole patching framework in the past.
> >
> > We do not have this experience, at least not as much as you seem to
> > have. Customer projects are developed completely in plugins, using the
> > framework and avoiding changes there. There ARE some changes tagged as
> > mods but not too much to be a real pain during a vendor update. Some
> > mods are also contributed back to OFBiz so these mods disappear during
> > the next vendor import.
> >
> > We are not directly using the original framework but rather use the
> > original as an upstream branch for our own projects. This makes it
> > easy to track changes and merge them into our codebase.
> >
> > It would be helpful to have some specific examples on where you are
> > having those painful changes.
> >
> >>
> >> _2. All or nothing:_In OFBiz, if you want to use the system, either
> >> take the whole thing or take nothing. What if I don't want any user
> >> interface for example, and just want to utilize the entities and
> >> services?
> >
> > We are running projects which do not use the original user interface
> > or are being headless / just REST interfaces. You are not forced to
> > use the UI and you can switch off the web apps, do we really have a
> > problem there?
> >
> >>
> >> _3. Limited Re-usability:_Because of the way current components are
> >> designed, it's difficult to take advantage of components in other
> >> components. Sharing artifacts is pretty much limited to entities and
> >> services. Even then they're not very shareable because they're
> >> designed in a non-usable way for a certain context and certain
> >> settings and database parameters.*
> >
> > Datamodel and services, the core business artifacts, are shareable and
> > in my view that's what's needed mostly. Can you be more specific
> > what's missing or designed in a way that it is not (re)usable ?
> >
> >
> >> *
> >>
> >> _4. Fragmented Actions:_**Instead of having unified XML actions, you
> >> have fragmentation across different contexts (screens, services,
> >> SECAs, etc). Furthermore, you cannot mix XML actions with groovy or
> >> other artifacts except by calling into external scripts and things
> >> like that. In comparison to moqui which is more flexible in XML
> >> actions I find it restrictive to work with OFBiz actions and it's
> >> hard to review, debug, or modify. I'm listing something below as an
> >> example for comparison [1]*
> >
> > I will have to investigate more how Moqui does those things. I'm not
> > sure what the example shows. Is it part of the controller logic?
> >
> > Generally, I think that it is a strength to have these tasks separated
> > in an MVC pattern and not being able to mix controller logic with
> > business logic.
> >
> > When it comes to debugging: how do you debug the shown XML code?
> >
> >
> >> *
> >>
> >> _5. Limited Routing capability:_**Routing is very minimal
> >> (controller.xml). It cannot contain business logic and is limited
> >> either to chaining requests or rendering screens. Sophisticated
> >> routing logic is important for making rich systems that react to
> >> differing circumstances.
> >> **
> >
> > See above: the controller part should NOT contain too much business
> > logic. If the business logic would be mixed with the controller logic,
> > how would you then make the business logic more shareable (see point 3
> > above)?
> >
> > Maybe I'm missing something and some examples would help me understand
> > the point.
> >
> >
> >>
> >> _6. Core framework difficult to change:_**OFBiz core was written
> >> a/long/time ago, and back then you didn't have many libraries for
> >> caching, transactions, DB pooling, Concurrency stuff, and others, so
> >> a lot was hand crafted, and we have a lots of interconnected API that
> >> is difficult to change, and most of these APIs depend on static util
> >> files that just kept piling and piling over the years. Where is the
> >> service engine implementation exactly? What about the entity engine?
> >> What about widgets? They are all over the place! Pieces for XML
> >> parsing, others for DOM models, and so on. The code is scattered and
> >> fixing it means going through a maze of many classes that are
> >> interconnected and all referencing each other. It's also hard to
> >> identify the signature or figure out the "big picture" since you have
> >> essentially minimal interfaces and no clear public API that you can
> >> refactor against. Pretty much the whole thing is public API**[2].
> >> **
> >
> > That's a big field and I think we should break this into subtopics to
> > discuss. Can you be more specific, for me this is too broad to
> > discuss. What is the real problem, what would be a better solution, ...?
> >
> > In general, a framework like OFBiz IS complex, it will never be easy
> > to read/understand. And of course, there is code which should be
> > refactored and better maintained (deprecation etc.). It would maybe be
> > a good start to identify those and do refactorings there first to
> > reduce complexity and pave the path for bigger refactorings?
> >
> > From my experience, e.g. the caching capabilities of OFBiz are a big
> > strength and to change those parts, we need some really good proof of
> > concepts to check if other solutions are really better. I've had
> > several bad experiences with, for example, object relational mapping
> > frameworks (performance, complexity) and I'm still a fan of the entity
> > engine.
> >
> >
> >>
> >> _7. Web-apps design problem__:_ A big problem (in my opinion) in the
> >> design is around having multiple web-apps, one for each component (or
> >> more). This interrupts the flow of the app and makes it difficult to
> >> come up with smooth solutions that work across the various screens
> >> and instead we need to pass a key between the components (and face
> >> other problems around session management). This restricts you in what
> >> you can offer as a user experience.*
> >> *
> >
> > Do you have a solution in mind to solve this and also keep those
> > applications separated (see above)? Single Sign On?
> >
> >
> >>
> >> _8. Problematic UI design__:_ Another/big/problem (in my opinion) is
> >> having the UI show everything (reference implementation). I never
> >> understood the value of showing/everything/! If I want to understand
> >> the data model, I will just look at the data model. The purpose of
> >> the UI is not to show everything, but to implement specific useful
> >> use-cases. The reference implementation we have is not doing that,
> >> and not offering something very useful, instead it is showing
> >> everything! To improve this, we need to switch from data-model based
> >> interface (party, order, etc ...) to use-case based interface
> >> (project management, e-commerce, ERP, CRM, etc ...)*
> >> *
> >
> > That's indeed a big issue and part of the discussions with every new
> > user who wants to utilize the existing web apps. On one hand it is
> > great to be able to show the potential of the ecosystem, on the other
> > hand it's too complex, especially for new users.
> >
> > IMO this is not so much a problem of the framework in general, which
> > does not prevent us from developing better UI and workflows.
> > Personally, I see OFBiz as an ERP framework which has to be customized
> > to match the target users's needs and not so much as an ready-to-use ERP.
> >
> > But yes, this can and should be improved with additional web apps or
> > even ONE web app which tightly integrates all core applications into a
> > one-stop ERP solution which is more user-friendly than the base
> > applications. On the other hand, this will make separation more
> > difficult...
> >
> >
> >>
> >> _9. Limited client rendering:_Because it is difficult to refactor the
> >> framework, it's hard to pin-point where to go exactly to refactor all
> >> the code needed to support more client-rendering technology (react,
> >> vue, angular, etc ...) and make it interact smoothly with the
> >> back-end (drop-downs, complex forms, etc ...). A lot of work needs to
> >> happen both to the /themes and to /framework to allow and provide
> >> more flexibility (also moving things to DB). If this step is not done
> >> you will be stuck with your UI capabilities and won't catch up
> >> (unless you develop your own UI which makes the UI of the community
> >> redundant and you let go of the whole widget system anyway).*
> >> *
> >
> > I guess it would help a lot if we have REST integrated into the core
> > framework and expose a proper REST API. For the HTML based web UI, I
> > have not much to change as the widget/freemarker templating gives us
> > much flexibility.
> >
> > What do you miss exactly / in which direction do you want to do changes?
> >
> >
> >>
> >> _10. Security is difficult to implement:_One of my favorite features
> >> in moqui is having security baked in the artifacts themselves to
> >> avoid sprinkling all screens / services with security logic. I never
> >> even worry about writing security related stuff until way close to
> >> the end of delivering the project. So there needs to be a whole
> >> security layer that sits below all the artifacts such that we don't
> >> even need to write security code in the various screens and services
> >> and around certain entities.*
> >> *
> >
> > Can you elaborate more on how you want to achieve fine grained
> > permissions for screens and services if not tied to the screen/service
> > configurations (having in mind that they should be reusable)?
> >
> >
> >>
> >> _11. Limited Integration Solutions__:_ I know we have now a REST
> >> component, but this is not the same as being a core feature of the
> >> framework (essential for any modern web framework). Why should it be
> >> tightly integrated? Many reasons: security, localization, mapping to
> >> service engine, mapping to entities, etc ...*
> >> *
> >
> > I'm currently working on the rest-api plugin and I agree that it would
> > be great to have it in the framework instead of being a plugin.
> >
> >
> >> _12. No Specialization:_Because of the big repository, and having
> >> everything point to everything, it's hard to specialize. Now compare
> >> this to the approach we have in our work which is easier to
> >> specialize in. For example, I have a component for payment gateway
> >> integration, another one for twilio (SMS integration), and I have
> >> application components for each solution in its own repository (HR,
> >> insurance stuff, many others) and each is living in its own git repo
> >> yet they work together just fine and without touching one line of
> >> code in the framework or even core components (despite depending on
> >> them and using pieces of them). I can jump into new versions of the
> >> framework everyday without breaking a sweat. When I work on component
> >> X, I'm only thinking of that and nothing else. I don't have to worry
> >> about changing something that would crash another component or bring
> >> the system down (happened to me many times in the past).
> >
> > For our projects, we were going the other way: we have framework +
> > plugins in the same repository instead of handling different
> > repositories. I think it would be a real pain to split the project in
> > dozens of different repositories only to bring them together to
> > integrate during development or deployment again.
> >
> > Can you describe how you are handling those different repositories in
> > an efficient way to get a better picture?
> >
> > As long as your work in a component/application/plugin and don't touch
> > the framework code I also do not see why you should break something
> > for other components? Of course, there ARE applications which are more
> > fundamental than other (e.g. party, security) and if those are changed
> > they might break other functionality. I see those applications
> > belonging to the framework more than being independent applications.
> >
> > I agree that we should try to identify false or unnecessary
> > dependencies though.
> >
> >
> >>
> >>
> >> Solutions
> >>
> >> Whatever solutions we come up with I think they're going to be
> >> difficult and would require community buy-in and support. I'm
> >> breaking this down into things to keep, update, add and remove as
> >> follows:
> >>
> >>
> >> Keep it
> >>
> >> - XSD signatures (so entity definitions, service definitions, XML
> >> actions though should expand)
> >> - Basic constructs of what is a component (with basic directory
> >> structure)
> >> - Data model
> >> - Existing services (core data-model services)
> >> - Java / groovy code that doesn't use core framework constructs or
> >> goes too deep into the internals.
> >> - Most used and useful APIs (Making queries, calling services, etc
> >> ...) although it should all be converted to interfaces with separate
> >> implementations where it's lacking.
> >>
> >
> > Agree to all above.
> >
> >
> >>
> >> Improve It
> >>
> >> - Break the project to smaller git repositories.
> >> - framework
> >> - data model
> >> - services
> >> - common UI
> >> - use-case applications (ERP, CRM, e-commerce, etc ...)
> >> - plugins
> >> - entire framework directory needs refurbishment and big chunks
> >> probably require a delete / rewrite
> >> - the web architecture (get rid of web-apps)
> >> - the tests (lots of them need to be deleted anyway)
> >> - Routing needs to be programmatic with ability to inject code.
> >>
> >>
> >> Add It
> >>
> >> - Configuration override mechanism to remove the need to touch the
> >> core. Overrides include everything including routing, component
> >> registration, etc ...
> >> - REST integration into the core
> >> - Artifact based security that maps users / groups to artifacts
> >> without the need to inject security code into every screen
> >> - A super-set of XML actions that can be applied in all contexts and
> >> can embed groovy code
> >> - New, simple use-case based user interface. It needs to be designed
> >> such that it is easy to extend to avoid the all-or-nothing UI that
> >> currently exists.
> >>
> >>
> >> Remove It / Deprecate It / Overwrite It
> >>
> >> - Most of the existing UI (needs to be replaced with simpler use-case
> >> based interface)
> >> - Lots of the core framework code (needs major rewrites, needs
> >> interfaces, import libraries for things like transactions, security,
> >> concurrency, etc ...)
> >> - Probably some of the existing plugins might need to be deprecated.
> >
> > See my questions above for some of the topics. I will add more details
> > and my point-of-view once we've split up the topics.
> >
> > The overall change would be huge and we should paint a clear picture
> > of what we want to achieve and which way to go in detail.
> >
> > We'll have to prioritize with a good cost/effort consideration and do
> > implementations step-by-step with a proper migration plan. Maybe
> > having feature branches for a longer time to be able to put out
> > releases if the development gets stuck or lasts longer.
> >
> >
> >>
> >>
> >> Quality Control
> >>
> >> Investing in such a huge effort requires that we change the way we
> >> view the project after / while work is being completed. Code should
> >> be approached with greater care and a lot of emphasis should be put
> >> on high quality, readability, avoiding anti-patterns, enforcing good
> >> patterns and best-practices. Some of the daily workflows need to
> >> improve / change, more thorough auditing and peer-reviews, and so on.
> >
> > A big +1 for quality control. This will be a real challenge for the
> > project as it requires dedication and enough ressources for review.
> >
> >
> >>
> >>
> >> Tradeoffs
> >>
> >> It goes without saying that the above is a _lot_ of work and
> >> commitment and requires a lot of effort. That's why I hesitated in
> >> starting this thread, but I thought maybe showing the tradeoffs would
> >> make it easier to weigh things out and decide on a direction to push.
> >>
> >>
> >> Pros
> >>
> >> - Pretty much most of the above mentioned problems will be resolved.
> >> - The framework would be resilient and future proof.
> >> - We will have more options and capabilities in the future, it would
> >> be easier to incorporate new technologies as they emerge
> >> - Development would go faster
> >> - Technical Debt will be reduced
> >>
> >>
> >> Cons
> >>
> >> - Huge (maybe too huge) upfront costs in time and effort to make the
> >> switch
> >> - Extra effort needed to ensure a smooth upgrade path for existing
> >> users / adopters / system integrators. A whole layer of code needs to
> >> be written to ensure the ability to upgrade without massive rewrites.
> >> - Risk of failure, exhaustion, inability to push through to the end
> >> - Inconvenience to those with existing systems / solutions
> >>
> >>
> >> Summary
> >>
> >> Change is the only constant. Inability to adapt to the future means
> >> not surviving. I'm not confident what is the best change possible or
> >> at what pace, but I do know that lack of change is a death sentence
> >> to any project. Maybe it's time to reflect and think of where we'd
> >> like to go and this thread is sort of a brain-dump of possible ideas.
> >>
> >> I look forward to hearing your thoughts.
> >
> > Thanks, Taher, for the thoughtful write-up of your view on the project
> > and the fields of change you see to be tackled. I do not agree with
> > all of them and I have a sensible view on cost/effort, thorough
> > planning and the need to always have existing users/projects in mind.
> > But I hope that we/the project will find a way to constantly work on
> > improvements.
> >
> > Again, I would propose to split up the most important topics into
> > threads with accompanying documentation in the wiki to help avoid
> > confusion.
> >
> > I am curious what others have to say and if we can activate enough
> > interest to improve.
> >
> >
> >>
> >> Cheers,
> >>
> >> Taher Alkhateeb
> >>
> >>
> >> [1] Example mixed XML actions and groovy from moqui
> >>
> >> <if condition="settlementTerm?.orderPmtServiceRegisterId">
> >> <then>
> >> <service-call name="update#mantle.order.OrderPart"
> >> in-map="[orderId:orderId,
> >> orderPartSeqId:orderPartSeqId, settlementTermId:settlementTermId]"/>
> >>
> >> <entity-find-one entity-name="moqui.service.ServiceRegister"
> >> value-field="serviceRegister">
> >> <field-map field-name="serviceRegisterId"
> >> from="settlementTerm.orderPmtServiceRegisterId"/>
> >> </entity-find-one>
> >> <script><![CDATA[
> >> try {
> >>
> ec.service.sync().name(serviceRegister.serviceName).parameters(context).softValidate(true).disableAuthz().call()
>
> >>
> >> if (ec.message.hasError()) return
> >> } catch (Throwable t) {
> >> ec.logger.log(300, "Error calling auto order payments
> >> service for SettlementTerm ${settlementTermId}", t)
> >> }
> >> ]]></script>
> >> </then>
> >> <else>
> >> <service-call
> >> name="mantle.account.PaymentServices.create#Payment" out-map="context"
> >> in-map="context + [amount:amount,
> >> amountUomId:amountUomId, fromPartyId:fromPartyId,
> >> toPartyId:toPartyId]"/>
> >> </else>
> >> </if>
> >>
> >> [2] https://ci.apache.org/projects/ofbiz/site/trunk/javadocs/
> >>
>
Re: OFBiz re-architecture thoughts
Posted by Taher Alkhateeb <ta...@pythys.com.INVALID>.
Hi Michael,
Thank you so much for spending your time reading and sifting through
everything, I really appreciate and value your feedback.
I need to highlight that we deployed mixed and somewhat crazy solutions
including multiple mobile apps, live video / chat streaming systems and
other projects that are booming in Kuwait and exploding both in user
bases and technical requirements which is why we cannot operate some of
these projects the way we used to in the past.
Anyway I think I'm not going to elaborate too much as we wait for more
opinions from the community to see if this is worth pushing. However, I
want to reply to a few points you raised:
- There is a huge difference between _few_ modifications, and ZERO
modifications to the framework or core components. This way you don't
even need to freeze a version of OFBiz to develop against at all!
- Breaking to multiple git repositories means having something like
./gradlew gitPullAll to update and you're done. Nothing breaks, no
surprises. You have to try it to believe it :) I have client projects
that have 15 git repos and I can upgrade with my eyes closed. Also there
is always entanglement that you don't feel or care for until you
actually cut the repo into pieces and then realize we have too much
entanglement. Having many small git repositories is in fact the most
enjoyable part of how we're writing software now and I can assign
different modules to different teams and they work independent of each
other. It's just a totally different ballgame.
- All or nothing in terms of code means you have to onboard a lot of
code that you don't need. I'm not saying you cannot it turn it off, I'm
saying it should be opt-in, not opt-out. All this code is potential
maintenance, bugs, security, and so on that you shouldn't carry on if
you don't want it. This makes things the messiest in terms of seed /
demo data but also in other entanglements.
- Yes entities and services are re-usable but you're limited to only
that. Powerful re-usability is applied to everything (configurations,
routers, widgets, scripts, etc ...). What is mostly re-usable in OFBiz
is entities and services, the others are mostly "usable" not "reusable".
It would take too long to explain and provide examples so I'll delay
this part.
- It's so awesome to use groovy exactly where you need it, without
having to jump into yet another file to only write 2 lines of code. I
have a service that needs 5-6 lines of code, why do I have to point to
an external script. Put the implementation RIGHT THERE next to the
definition of your service for example, the same is true everywhere you
need XML actions. So scripting in lacking in XML actions, and also they
are inconsistent (works differently in different places).
- Mixing XML with groovy is just part of the flexibility and having your
code where you need it. Even debugging XML code improves by putting
groovy snippets above / below to investigate the context. Nobody would
want to do that by opening a new file just to print something from the
context for example.
- Services are pure business logic, controller is routing logic. Routing
logic does not mean NO logic, it means logic that is focused on
coordination and decisions on where to go and what to do and putting
that logic in services is actually mixing layers and making the code
harder to understand / debug. I cannot show you how severely limited
routing logic is in OFBiz is without showing you 5 / 6 examples of
powerful things we've done that we simply can not do in OFBiz.
- We should IMHO by default have ONLY one web-app with the ability to
add more in an "opt-in" fashion. However, this means we need to be able
to break the screen definitions and routes, routing needs a major change
to achieve this objective. The controller.xml would simply not cut it.
Again I need examples to really highlight what I mean and I might do it
later.
- It is not enough to integrate REST to be able to have dynamic
libraries integrated with OFBiz if you want that to work with your
widgets and other constructs. You also need multiple rendering modes
that allow you to select from a variety of different technologies /
options with different macros and different theming constructs. Again I
cannot say more without examples to illustrate clearly.
- Finally, many of the changes that I'm proposing require substantial
changes to the framework and redesigning it. Well we cannot do that
without first refactoring it, which means we have to get rid of / update
lots of code. which means we need to start using libraries to control
the large size of that thing. It's a chain, one thing pulling another
thing, which makes it inevitable that a major shakeup of the core
framework is necessary. I'm not proposing to change it just to change
it. I'm proposing changing it because without that many of the things
I'm proposing simply won't happen.
I can understand why you might not agree with some of the points. In a
way it's kind of hard to imagine a different way of doing things when
we're used to working a certain way, but once we get exposed we realize
that OH .. I've been torturing myself :). However, I do so your
very-good point on being conservative because there is substantial risk
in doing everything that I suggest.
Anyway opening this to the rest of the community and thank you in
advance for sharing your thoughts.
Cheers,
Taher Alkhateeb
On 12/27/21 17:20, Michael Brohl wrote:
> Hi Taher,
>
> thanks for bringing this up! I will try to briefly explain my views on
> the raised topics.
>
> My perspective is from nearly 20 years of intensive work with OFBiz,
> both for internal and external projects. Those projects have/had a
> volume reaching from about 50 to over 1000 project days and we have
> active customer projects which are running for almost 20 years now,
> still being developed further with the business. OFBiz often became a
> crucial part of the business ecosystem of those users. It is also the
> central toolbox for our services around development of web based
> software solutions and system integration in our own company since 7
> years now. I consider myself both a user of OFBiz and a service
> provider/integrator.
>
> After so many years of working with OFBiz if have learned that - from
> a business users perspective - the functionality and integration
> aspects along with the powerful datamodel, services etc. as well as
> flexibility, performance and scalability is way more important than
> detailed technical aspects (in the sense of “modern” or “newest
> technology”).
>
> As a service provider for OFBiz, we are feeling responsible to provide
> a reliable, constant and upgradeable software ecosystem which is why
> we are on a more "conservative" path regarding big or long running
> changes in the project. I have seen several new, ambititious
> contributors who wanted to change everything, using the newest
> frameworks and tools around etc., but often did not have an extensive
> background of real world projects or who are not able to explain why
> they want to use other tools and mechanisms and which user problems
> they really want to solve.
>
> Most of these people are not active anymore or unfinished solutions
> are put into the codebase which is another reason why I am really
> careful when it comes to massive changes. I think the project needs
> good planning, a good migration path and constant support when those
> changes are decided to take place.
>
> All in all, I do not feel such a great pain as you seem to feel but
> there is definetely much room for improvement, for sure.
>
> I will try to give my views on the different topics inline. They are
> brief and need/deserve more in-depth preparation from my side which I
> cannot spare at the moment.
>
> For further discussions I recommend to split those topics in different
> threads because they would become unreadable soon with more
> participants. Also writing up the outcome of the discussions and the
> further roadmap in the wiki might help.
>
> Let's see how it works.
>
> Michael Brohl
>
> ecomify GmbH - www.ecomify.de
>
>
> Am 22.12.21 um 18:23 schrieb Taher Alkhateeb:
>> Problems
> ...
>> _1. Lack of Code Isolation_:It is unlikely to do anything substantial
>> in OFBiz without touching the core repositories. Compare this with
>> moqui or some other web frameworks like say Ruby-On-Rails or django
>> and you will see that you can work independently of the development
>> path in these frameworks. This makes upgrades quite painfulwith
>> patches and manual interventions. Some in the community even
>> introduced a whole patching framework in the past.
>
> We do not have this experience, at least not as much as you seem to
> have. Customer projects are developed completely in plugins, using the
> framework and avoiding changes there. There ARE some changes tagged as
> mods but not too much to be a real pain during a vendor update. Some
> mods are also contributed back to OFBiz so these mods disappear during
> the next vendor import.
>
> We are not directly using the original framework but rather use the
> original as an upstream branch for our own projects. This makes it
> easy to track changes and merge them into our codebase.
>
> It would be helpful to have some specific examples on where you are
> having those painful changes.
>
>>
>> _2. All or nothing:_In OFBiz, if you want to use the system, either
>> take the whole thing or take nothing. What if I don't want any user
>> interface for example, and just want to utilize the entities and
>> services?
>
> We are running projects which do not use the original user interface
> or are being headless / just REST interfaces. You are not forced to
> use the UI and you can switch off the web apps, do we really have a
> problem there?
>
>>
>> _3. Limited Re-usability:_Because of the way current components are
>> designed, it's difficult to take advantage of components in other
>> components. Sharing artifacts is pretty much limited to entities and
>> services. Even then they're not very shareable because they're
>> designed in a non-usable way for a certain context and certain
>> settings and database parameters.*
>
> Datamodel and services, the core business artifacts, are shareable and
> in my view that's what's needed mostly. Can you be more specific
> what's missing or designed in a way that it is not (re)usable ?
>
>
>> *
>>
>> _4. Fragmented Actions:_**Instead of having unified XML actions, you
>> have fragmentation across different contexts (screens, services,
>> SECAs, etc). Furthermore, you cannot mix XML actions with groovy or
>> other artifacts except by calling into external scripts and things
>> like that. In comparison to moqui which is more flexible in XML
>> actions I find it restrictive to work with OFBiz actions and it's
>> hard to review, debug, or modify. I'm listing something below as an
>> example for comparison [1]*
>
> I will have to investigate more how Moqui does those things. I'm not
> sure what the example shows. Is it part of the controller logic?
>
> Generally, I think that it is a strength to have these tasks separated
> in an MVC pattern and not being able to mix controller logic with
> business logic.
>
> When it comes to debugging: how do you debug the shown XML code?
>
>
>> *
>>
>> _5. Limited Routing capability:_**Routing is very minimal
>> (controller.xml). It cannot contain business logic and is limited
>> either to chaining requests or rendering screens. Sophisticated
>> routing logic is important for making rich systems that react to
>> differing circumstances.
>> **
>
> See above: the controller part should NOT contain too much business
> logic. If the business logic would be mixed with the controller logic,
> how would you then make the business logic more shareable (see point 3
> above)?
>
> Maybe I'm missing something and some examples would help me understand
> the point.
>
>
>>
>> _6. Core framework difficult to change:_**OFBiz core was written
>> a/long/time ago, and back then you didn't have many libraries for
>> caching, transactions, DB pooling, Concurrency stuff, and others, so
>> a lot was hand crafted, and we have a lots of interconnected API that
>> is difficult to change, and most of these APIs depend on static util
>> files that just kept piling and piling over the years. Where is the
>> service engine implementation exactly? What about the entity engine?
>> What about widgets? They are all over the place! Pieces for XML
>> parsing, others for DOM models, and so on. The code is scattered and
>> fixing it means going through a maze of many classes that are
>> interconnected and all referencing each other. It's also hard to
>> identify the signature or figure out the "big picture" since you have
>> essentially minimal interfaces and no clear public API that you can
>> refactor against. Pretty much the whole thing is public API**[2].
>> **
>
> That's a big field and I think we should break this into subtopics to
> discuss. Can you be more specific, for me this is too broad to
> discuss. What is the real problem, what would be a better solution, ...?
>
> In general, a framework like OFBiz IS complex, it will never be easy
> to read/understand. And of course, there is code which should be
> refactored and better maintained (deprecation etc.). It would maybe be
> a good start to identify those and do refactorings there first to
> reduce complexity and pave the path for bigger refactorings?
>
> From my experience, e.g. the caching capabilities of OFBiz are a big
> strength and to change those parts, we need some really good proof of
> concepts to check if other solutions are really better. I've had
> several bad experiences with, for example, object relational mapping
> frameworks (performance, complexity) and I'm still a fan of the entity
> engine.
>
>
>>
>> _7. Web-apps design problem__:_ A big problem (in my opinion) in the
>> design is around having multiple web-apps, one for each component (or
>> more). This interrupts the flow of the app and makes it difficult to
>> come up with smooth solutions that work across the various screens
>> and instead we need to pass a key between the components (and face
>> other problems around session management). This restricts you in what
>> you can offer as a user experience.*
>> *
>
> Do you have a solution in mind to solve this and also keep those
> applications separated (see above)? Single Sign On?
>
>
>>
>> _8. Problematic UI design__:_ Another/big/problem (in my opinion) is
>> having the UI show everything (reference implementation). I never
>> understood the value of showing/everything/! If I want to understand
>> the data model, I will just look at the data model. The purpose of
>> the UI is not to show everything, but to implement specific useful
>> use-cases. The reference implementation we have is not doing that,
>> and not offering something very useful, instead it is showing
>> everything! To improve this, we need to switch from data-model based
>> interface (party, order, etc ...) to use-case based interface
>> (project management, e-commerce, ERP, CRM, etc ...)*
>> *
>
> That's indeed a big issue and part of the discussions with every new
> user who wants to utilize the existing web apps. On one hand it is
> great to be able to show the potential of the ecosystem, on the other
> hand it's too complex, especially for new users.
>
> IMO this is not so much a problem of the framework in general, which
> does not prevent us from developing better UI and workflows.
> Personally, I see OFBiz as an ERP framework which has to be customized
> to match the target users's needs and not so much as an ready-to-use ERP.
>
> But yes, this can and should be improved with additional web apps or
> even ONE web app which tightly integrates all core applications into a
> one-stop ERP solution which is more user-friendly than the base
> applications. On the other hand, this will make separation more
> difficult...
>
>
>>
>> _9. Limited client rendering:_Because it is difficult to refactor the
>> framework, it's hard to pin-point where to go exactly to refactor all
>> the code needed to support more client-rendering technology (react,
>> vue, angular, etc ...) and make it interact smoothly with the
>> back-end (drop-downs, complex forms, etc ...). A lot of work needs to
>> happen both to the /themes and to /framework to allow and provide
>> more flexibility (also moving things to DB). If this step is not done
>> you will be stuck with your UI capabilities and won't catch up
>> (unless you develop your own UI which makes the UI of the community
>> redundant and you let go of the whole widget system anyway).*
>> *
>
> I guess it would help a lot if we have REST integrated into the core
> framework and expose a proper REST API. For the HTML based web UI, I
> have not much to change as the widget/freemarker templating gives us
> much flexibility.
>
> What do you miss exactly / in which direction do you want to do changes?
>
>
>>
>> _10. Security is difficult to implement:_One of my favorite features
>> in moqui is having security baked in the artifacts themselves to
>> avoid sprinkling all screens / services with security logic. I never
>> even worry about writing security related stuff until way close to
>> the end of delivering the project. So there needs to be a whole
>> security layer that sits below all the artifacts such that we don't
>> even need to write security code in the various screens and services
>> and around certain entities.*
>> *
>
> Can you elaborate more on how you want to achieve fine grained
> permissions for screens and services if not tied to the screen/service
> configurations (having in mind that they should be reusable)?
>
>
>>
>> _11. Limited Integration Solutions__:_ I know we have now a REST
>> component, but this is not the same as being a core feature of the
>> framework (essential for any modern web framework). Why should it be
>> tightly integrated? Many reasons: security, localization, mapping to
>> service engine, mapping to entities, etc ...*
>> *
>
> I'm currently working on the rest-api plugin and I agree that it would
> be great to have it in the framework instead of being a plugin.
>
>
>> _12. No Specialization:_Because of the big repository, and having
>> everything point to everything, it's hard to specialize. Now compare
>> this to the approach we have in our work which is easier to
>> specialize in. For example, I have a component for payment gateway
>> integration, another one for twilio (SMS integration), and I have
>> application components for each solution in its own repository (HR,
>> insurance stuff, many others) and each is living in its own git repo
>> yet they work together just fine and without touching one line of
>> code in the framework or even core components (despite depending on
>> them and using pieces of them). I can jump into new versions of the
>> framework everyday without breaking a sweat. When I work on component
>> X, I'm only thinking of that and nothing else. I don't have to worry
>> about changing something that would crash another component or bring
>> the system down (happened to me many times in the past).
>
> For our projects, we were going the other way: we have framework +
> plugins in the same repository instead of handling different
> repositories. I think it would be a real pain to split the project in
> dozens of different repositories only to bring them together to
> integrate during development or deployment again.
>
> Can you describe how you are handling those different repositories in
> an efficient way to get a better picture?
>
> As long as your work in a component/application/plugin and don't touch
> the framework code I also do not see why you should break something
> for other components? Of course, there ARE applications which are more
> fundamental than other (e.g. party, security) and if those are changed
> they might break other functionality. I see those applications
> belonging to the framework more than being independent applications.
>
> I agree that we should try to identify false or unnecessary
> dependencies though.
>
>
>>
>>
>> Solutions
>>
>> Whatever solutions we come up with I think they're going to be
>> difficult and would require community buy-in and support. I'm
>> breaking this down into things to keep, update, add and remove as
>> follows:
>>
>>
>> Keep it
>>
>> - XSD signatures (so entity definitions, service definitions, XML
>> actions though should expand)
>> - Basic constructs of what is a component (with basic directory
>> structure)
>> - Data model
>> - Existing services (core data-model services)
>> - Java / groovy code that doesn't use core framework constructs or
>> goes too deep into the internals.
>> - Most used and useful APIs (Making queries, calling services, etc
>> ...) although it should all be converted to interfaces with separate
>> implementations where it's lacking.
>>
>
> Agree to all above.
>
>
>>
>> Improve It
>>
>> - Break the project to smaller git repositories.
>> - framework
>> - data model
>> - services
>> - common UI
>> - use-case applications (ERP, CRM, e-commerce, etc ...)
>> - plugins
>> - entire framework directory needs refurbishment and big chunks
>> probably require a delete / rewrite
>> - the web architecture (get rid of web-apps)
>> - the tests (lots of them need to be deleted anyway)
>> - Routing needs to be programmatic with ability to inject code.
>>
>>
>> Add It
>>
>> - Configuration override mechanism to remove the need to touch the
>> core. Overrides include everything including routing, component
>> registration, etc ...
>> - REST integration into the core
>> - Artifact based security that maps users / groups to artifacts
>> without the need to inject security code into every screen
>> - A super-set of XML actions that can be applied in all contexts and
>> can embed groovy code
>> - New, simple use-case based user interface. It needs to be designed
>> such that it is easy to extend to avoid the all-or-nothing UI that
>> currently exists.
>>
>>
>> Remove It / Deprecate It / Overwrite It
>>
>> - Most of the existing UI (needs to be replaced with simpler use-case
>> based interface)
>> - Lots of the core framework code (needs major rewrites, needs
>> interfaces, import libraries for things like transactions, security,
>> concurrency, etc ...)
>> - Probably some of the existing plugins might need to be deprecated.
>
> See my questions above for some of the topics. I will add more details
> and my point-of-view once we've split up the topics.
>
> The overall change would be huge and we should paint a clear picture
> of what we want to achieve and which way to go in detail.
>
> We'll have to prioritize with a good cost/effort consideration and do
> implementations step-by-step with a proper migration plan. Maybe
> having feature branches for a longer time to be able to put out
> releases if the development gets stuck or lasts longer.
>
>
>>
>>
>> Quality Control
>>
>> Investing in such a huge effort requires that we change the way we
>> view the project after / while work is being completed. Code should
>> be approached with greater care and a lot of emphasis should be put
>> on high quality, readability, avoiding anti-patterns, enforcing good
>> patterns and best-practices. Some of the daily workflows need to
>> improve / change, more thorough auditing and peer-reviews, and so on.
>
> A big +1 for quality control. This will be a real challenge for the
> project as it requires dedication and enough ressources for review.
>
>
>>
>>
>> Tradeoffs
>>
>> It goes without saying that the above is a _lot_ of work and
>> commitment and requires a lot of effort. That's why I hesitated in
>> starting this thread, but I thought maybe showing the tradeoffs would
>> make it easier to weigh things out and decide on a direction to push.
>>
>>
>> Pros
>>
>> - Pretty much most of the above mentioned problems will be resolved.
>> - The framework would be resilient and future proof.
>> - We will have more options and capabilities in the future, it would
>> be easier to incorporate new technologies as they emerge
>> - Development would go faster
>> - Technical Debt will be reduced
>>
>>
>> Cons
>>
>> - Huge (maybe too huge) upfront costs in time and effort to make the
>> switch
>> - Extra effort needed to ensure a smooth upgrade path for existing
>> users / adopters / system integrators. A whole layer of code needs to
>> be written to ensure the ability to upgrade without massive rewrites.
>> - Risk of failure, exhaustion, inability to push through to the end
>> - Inconvenience to those with existing systems / solutions
>>
>>
>> Summary
>>
>> Change is the only constant. Inability to adapt to the future means
>> not surviving. I'm not confident what is the best change possible or
>> at what pace, but I do know that lack of change is a death sentence
>> to any project. Maybe it's time to reflect and think of where we'd
>> like to go and this thread is sort of a brain-dump of possible ideas.
>>
>> I look forward to hearing your thoughts.
>
> Thanks, Taher, for the thoughtful write-up of your view on the project
> and the fields of change you see to be tackled. I do not agree with
> all of them and I have a sensible view on cost/effort, thorough
> planning and the need to always have existing users/projects in mind.
> But I hope that we/the project will find a way to constantly work on
> improvements.
>
> Again, I would propose to split up the most important topics into
> threads with accompanying documentation in the wiki to help avoid
> confusion.
>
> I am curious what others have to say and if we can activate enough
> interest to improve.
>
>
>>
>> Cheers,
>>
>> Taher Alkhateeb
>>
>>
>> [1] Example mixed XML actions and groovy from moqui
>>
>> <if condition="settlementTerm?.orderPmtServiceRegisterId">
>> <then>
>> <service-call name="update#mantle.order.OrderPart"
>> in-map="[orderId:orderId,
>> orderPartSeqId:orderPartSeqId, settlementTermId:settlementTermId]"/>
>>
>> <entity-find-one entity-name="moqui.service.ServiceRegister"
>> value-field="serviceRegister">
>> <field-map field-name="serviceRegisterId"
>> from="settlementTerm.orderPmtServiceRegisterId"/>
>> </entity-find-one>
>> <script><![CDATA[
>> try {
>> ec.service.sync().name(serviceRegister.serviceName).parameters(context).softValidate(true).disableAuthz().call()
>>
>> if (ec.message.hasError()) return
>> } catch (Throwable t) {
>> ec.logger.log(300, "Error calling auto order payments
>> service for SettlementTerm ${settlementTermId}", t)
>> }
>> ]]></script>
>> </then>
>> <else>
>> <service-call
>> name="mantle.account.PaymentServices.create#Payment" out-map="context"
>> in-map="context + [amount:amount,
>> amountUomId:amountUomId, fromPartyId:fromPartyId,
>> toPartyId:toPartyId]"/>
>> </else>
>> </if>
>>
>> [2] https://ci.apache.org/projects/ofbiz/site/trunk/javadocs/
>>
Re: OFBiz re-architecture thoughts
Posted by Michael Brohl <mi...@ecomify.de>.
Hi Taher,
thanks for bringing this up! I will try to briefly explain my views on
the raised topics.
My perspective is from nearly 20 years of intensive work with OFBiz,
both for internal and external projects. Those projects have/had a
volume reaching from about 50 to over 1000 project days and we have
active customer projects which are running for almost 20 years now,
still being developed further with the business. OFBiz often became a
crucial part of the business ecosystem of those users. It is also the
central toolbox for our services around development of web based
software solutions and system integration in our own company since 7
years now. I consider myself both a user of OFBiz and a service
provider/integrator.
After so many years of working with OFBiz if have learned that - from a
business users perspective - the functionality and integration aspects
along with the powerful datamodel, services etc. as well as flexibility,
performance and scalability is way more important than detailed
technical aspects (in the sense of “modern” or “newest technology”).
As a service provider for OFBiz, we are feeling responsible to provide a
reliable, constant and upgradeable software ecosystem which is why we
are on a more "conservative" path regarding big or long running changes
in the project. I have seen several new, ambititious contributors who
wanted to change everything, using the newest frameworks and tools
around etc., but often did not have an extensive background of real
world projects or who are not able to explain why they want to use other
tools and mechanisms and which user problems they really want to solve.
Most of these people are not active anymore or unfinished solutions are
put into the codebase which is another reason why I am really careful
when it comes to massive changes. I think the project needs good
planning, a good migration path and constant support when those changes
are decided to take place.
All in all, I do not feel such a great pain as you seem to feel but
there is definetely much room for improvement, for sure.
I will try to give my views on the different topics inline. They are
brief and need/deserve more in-depth preparation from my side which I
cannot spare at the moment.
For further discussions I recommend to split those topics in different
threads because they would become unreadable soon with more
participants. Also writing up the outcome of the discussions and the
further roadmap in the wiki might help.
Let's see how it works.
Michael Brohl
ecomify GmbH - www.ecomify.de
Am 22.12.21 um 18:23 schrieb Taher Alkhateeb:
> Problems
...
> _1. Lack of Code Isolation_:It is unlikely to do anything substantial
> in OFBiz without touching the core repositories. Compare this with
> moqui or some other web frameworks like say Ruby-On-Rails or django
> and you will see that you can work independently of the development
> path in these frameworks. This makes upgrades quite painfulwith
> patches and manual interventions. Some in the community even
> introduced a whole patching framework in the past.
We do not have this experience, at least not as much as you seem to
have. Customer projects are developed completely in plugins, using the
framework and avoiding changes there. There ARE some changes tagged as
mods but not too much to be a real pain during a vendor update. Some
mods are also contributed back to OFBiz so these mods disappear during
the next vendor import.
We are not directly using the original framework but rather use the
original as an upstream branch for our own projects. This makes it easy
to track changes and merge them into our codebase.
It would be helpful to have some specific examples on where you are
having those painful changes.
>
> _2. All or nothing:_In OFBiz, if you want to use the system, either
> take the whole thing or take nothing. What if I don't want any user
> interface for example, and just want to utilize the entities and
> services?
We are running projects which do not use the original user interface or
are being headless / just REST interfaces. You are not forced to use the
UI and you can switch off the web apps, do we really have a problem there?
>
> _3. Limited Re-usability:_Because of the way current components are
> designed, it's difficult to take advantage of components in other
> components. Sharing artifacts is pretty much limited to entities and
> services. Even then they're not very shareable because they're
> designed in a non-usable way for a certain context and certain
> settings and database parameters.*
Datamodel and services, the core business artifacts, are shareable and
in my view that's what's needed mostly. Can you be more specific what's
missing or designed in a way that it is not (re)usable ?
> *
>
> _4. Fragmented Actions:_**Instead of having unified XML actions, you
> have fragmentation across different contexts (screens, services,
> SECAs, etc). Furthermore, you cannot mix XML actions with groovy or
> other artifacts except by calling into external scripts and things
> like that. In comparison to moqui which is more flexible in XML
> actions I find it restrictive to work with OFBiz actions and it's hard
> to review, debug, or modify. I'm listing something below as an example
> for comparison [1]*
I will have to investigate more how Moqui does those things. I'm not
sure what the example shows. Is it part of the controller logic?
Generally, I think that it is a strength to have these tasks separated
in an MVC pattern and not being able to mix controller logic with
business logic.
When it comes to debugging: how do you debug the shown XML code?
> *
>
> _5. Limited Routing capability:_**Routing is very minimal
> (controller.xml). It cannot contain business logic and is limited
> either to chaining requests or rendering screens. Sophisticated
> routing logic is important for making rich systems that react to
> differing circumstances.
> **
See above: the controller part should NOT contain too much business
logic. If the business logic would be mixed with the controller logic,
how would you then make the business logic more shareable (see point 3
above)?
Maybe I'm missing something and some examples would help me understand
the point.
>
> _6. Core framework difficult to change:_**OFBiz core was written
> a/long/time ago, and back then you didn't have many libraries for
> caching, transactions, DB pooling, Concurrency stuff, and others, so a
> lot was hand crafted, and we have a lots of interconnected API that is
> difficult to change, and most of these APIs depend on static util
> files that just kept piling and piling over the years. Where is the
> service engine implementation exactly? What about the entity engine?
> What about widgets? They are all over the place! Pieces for XML
> parsing, others for DOM models, and so on. The code is scattered and
> fixing it means going through a maze of many classes that are
> interconnected and all referencing each other. It's also hard to
> identify the signature or figure out the "big picture" since you have
> essentially minimal interfaces and no clear public API that you can
> refactor against. Pretty much the whole thing is public API**[2].
> **
That's a big field and I think we should break this into subtopics to
discuss. Can you be more specific, for me this is too broad to discuss.
What is the real problem, what would be a better solution, ...?
In general, a framework like OFBiz IS complex, it will never be easy to
read/understand. And of course, there is code which should be refactored
and better maintained (deprecation etc.). It would maybe be a good start
to identify those and do refactorings there first to reduce complexity
and pave the path for bigger refactorings?
From my experience, e.g. the caching capabilities of OFBiz are a big
strength and to change those parts, we need some really good proof of
concepts to check if other solutions are really better. I've had several
bad experiences with, for example, object relational mapping frameworks
(performance, complexity) and I'm still a fan of the entity engine.
>
> _7. Web-apps design problem__:_ A big problem (in my opinion) in the
> design is around having multiple web-apps, one for each component (or
> more). This interrupts the flow of the app and makes it difficult to
> come up with smooth solutions that work across the various screens and
> instead we need to pass a key between the components (and face other
> problems around session management). This restricts you in what you
> can offer as a user experience.*
> *
Do you have a solution in mind to solve this and also keep those
applications separated (see above)? Single Sign On?
>
> _8. Problematic UI design__:_ Another/big/problem (in my opinion) is
> having the UI show everything (reference implementation). I never
> understood the value of showing/everything/! If I want to understand
> the data model, I will just look at the data model. The purpose of the
> UI is not to show everything, but to implement specific useful
> use-cases. The reference implementation we have is not doing that, and
> not offering something very useful, instead it is showing everything!
> To improve this, we need to switch from data-model based interface
> (party, order, etc ...) to use-case based interface (project
> management, e-commerce, ERP, CRM, etc ...)*
> *
That's indeed a big issue and part of the discussions with every new
user who wants to utilize the existing web apps. On one hand it is great
to be able to show the potential of the ecosystem, on the other hand
it's too complex, especially for new users.
IMO this is not so much a problem of the framework in general, which
does not prevent us from developing better UI and workflows. Personally,
I see OFBiz as an ERP framework which has to be customized to match the
target users's needs and not so much as an ready-to-use ERP.
But yes, this can and should be improved with additional web apps or
even ONE web app which tightly integrates all core applications into a
one-stop ERP solution which is more user-friendly than the base
applications. On the other hand, this will make separation more difficult...
>
> _9. Limited client rendering:_Because it is difficult to refactor the
> framework, it's hard to pin-point where to go exactly to refactor all
> the code needed to support more client-rendering technology (react,
> vue, angular, etc ...) and make it interact smoothly with the back-end
> (drop-downs, complex forms, etc ...). A lot of work needs to happen
> both to the /themes and to /framework to allow and provide more
> flexibility (also moving things to DB). If this step is not done you
> will be stuck with your UI capabilities and won't catch up (unless you
> develop your own UI which makes the UI of the community redundant and
> you let go of the whole widget system anyway).*
> *
I guess it would help a lot if we have REST integrated into the core
framework and expose a proper REST API. For the HTML based web UI, I
have not much to change as the widget/freemarker templating gives us
much flexibility.
What do you miss exactly / in which direction do you want to do changes?
>
> _10. Security is difficult to implement:_One of my favorite features
> in moqui is having security baked in the artifacts themselves to avoid
> sprinkling all screens / services with security logic. I never even
> worry about writing security related stuff until way close to the end
> of delivering the project. So there needs to be a whole security layer
> that sits below all the artifacts such that we don't even need to
> write security code in the various screens and services and around
> certain entities.*
> *
Can you elaborate more on how you want to achieve fine grained
permissions for screens and services if not tied to the screen/service
configurations (having in mind that they should be reusable)?
>
> _11. Limited Integration Solutions__:_ I know we have now a REST
> component, but this is not the same as being a core feature of the
> framework (essential for any modern web framework). Why should it be
> tightly integrated? Many reasons: security, localization, mapping to
> service engine, mapping to entities, etc ...*
> *
I'm currently working on the rest-api plugin and I agree that it would
be great to have it in the framework instead of being a plugin.
> _12. No Specialization:_Because of the big repository, and having
> everything point to everything, it's hard to specialize. Now compare
> this to the approach we have in our work which is easier to specialize
> in. For example, I have a component for payment gateway integration,
> another one for twilio (SMS integration), and I have application
> components for each solution in its own repository (HR, insurance
> stuff, many others) and each is living in its own git repo yet they
> work together just fine and without touching one line of code in the
> framework or even core components (despite depending on them and using
> pieces of them). I can jump into new versions of the framework
> everyday without breaking a sweat. When I work on component X, I'm
> only thinking of that and nothing else. I don't have to worry about
> changing something that would crash another component or bring the
> system down (happened to me many times in the past).
For our projects, we were going the other way: we have framework +
plugins in the same repository instead of handling different
repositories. I think it would be a real pain to split the project in
dozens of different repositories only to bring them together to
integrate during development or deployment again.
Can you describe how you are handling those different repositories in an
efficient way to get a better picture?
As long as your work in a component/application/plugin and don't touch
the framework code I also do not see why you should break something for
other components? Of course, there ARE applications which are more
fundamental than other (e.g. party, security) and if those are changed
they might break other functionality. I see those applications belonging
to the framework more than being independent applications.
I agree that we should try to identify false or unnecessary dependencies
though.
>
>
> Solutions
>
> Whatever solutions we come up with I think they're going to be
> difficult and would require community buy-in and support. I'm breaking
> this down into things to keep, update, add and remove as follows:
>
>
> Keep it
>
> - XSD signatures (so entity definitions, service definitions, XML
> actions though should expand)
> - Basic constructs of what is a component (with basic directory
> structure)
> - Data model
> - Existing services (core data-model services)
> - Java / groovy code that doesn't use core framework constructs or
> goes too deep into the internals.
> - Most used and useful APIs (Making queries, calling services, etc
> ...) although it should all be converted to interfaces with separate
> implementations where it's lacking.
>
Agree to all above.
>
> Improve It
>
> - Break the project to smaller git repositories.
> - framework
> - data model
> - services
> - common UI
> - use-case applications (ERP, CRM, e-commerce, etc ...)
> - plugins
> - entire framework directory needs refurbishment and big chunks
> probably require a delete / rewrite
> - the web architecture (get rid of web-apps)
> - the tests (lots of them need to be deleted anyway)
> - Routing needs to be programmatic with ability to inject code.
>
>
> Add It
>
> - Configuration override mechanism to remove the need to touch the
> core. Overrides include everything including routing, component
> registration, etc ...
> - REST integration into the core
> - Artifact based security that maps users / groups to artifacts
> without the need to inject security code into every screen
> - A super-set of XML actions that can be applied in all contexts and
> can embed groovy code
> - New, simple use-case based user interface. It needs to be designed
> such that it is easy to extend to avoid the all-or-nothing UI that
> currently exists.
>
>
> Remove It / Deprecate It / Overwrite It
>
> - Most of the existing UI (needs to be replaced with simpler use-case
> based interface)
> - Lots of the core framework code (needs major rewrites, needs
> interfaces, import libraries for things like transactions, security,
> concurrency, etc ...)
> - Probably some of the existing plugins might need to be deprecated.
See my questions above for some of the topics. I will add more details
and my point-of-view once we've split up the topics.
The overall change would be huge and we should paint a clear picture of
what we want to achieve and which way to go in detail.
We'll have to prioritize with a good cost/effort consideration and do
implementations step-by-step with a proper migration plan. Maybe having
feature branches for a longer time to be able to put out releases if the
development gets stuck or lasts longer.
>
>
> Quality Control
>
> Investing in such a huge effort requires that we change the way we
> view the project after / while work is being completed. Code should be
> approached with greater care and a lot of emphasis should be put on
> high quality, readability, avoiding anti-patterns, enforcing good
> patterns and best-practices. Some of the daily workflows need to
> improve / change, more thorough auditing and peer-reviews, and so on.
A big +1 for quality control. This will be a real challenge for the
project as it requires dedication and enough ressources for review.
>
>
> Tradeoffs
>
> It goes without saying that the above is a _lot_ of work and
> commitment and requires a lot of effort. That's why I hesitated in
> starting this thread, but I thought maybe showing the tradeoffs would
> make it easier to weigh things out and decide on a direction to push.
>
>
> Pros
>
> - Pretty much most of the above mentioned problems will be resolved.
> - The framework would be resilient and future proof.
> - We will have more options and capabilities in the future, it would
> be easier to incorporate new technologies as they emerge
> - Development would go faster
> - Technical Debt will be reduced
>
>
> Cons
>
> - Huge (maybe too huge) upfront costs in time and effort to make the
> switch
> - Extra effort needed to ensure a smooth upgrade path for existing
> users / adopters / system integrators. A whole layer of code needs to
> be written to ensure the ability to upgrade without massive rewrites.
> - Risk of failure, exhaustion, inability to push through to the end
> - Inconvenience to those with existing systems / solutions
>
>
> Summary
>
> Change is the only constant. Inability to adapt to the future means
> not surviving. I'm not confident what is the best change possible or
> at what pace, but I do know that lack of change is a death sentence to
> any project. Maybe it's time to reflect and think of where we'd like
> to go and this thread is sort of a brain-dump of possible ideas.
>
> I look forward to hearing your thoughts.
Thanks, Taher, for the thoughtful write-up of your view on the project
and the fields of change you see to be tackled. I do not agree with all
of them and I have a sensible view on cost/effort, thorough planning and
the need to always have existing users/projects in mind. But I hope that
we/the project will find a way to constantly work on improvements.
Again, I would propose to split up the most important topics into
threads with accompanying documentation in the wiki to help avoid confusion.
I am curious what others have to say and if we can activate enough
interest to improve.
>
> Cheers,
>
> Taher Alkhateeb
>
>
> [1] Example mixed XML actions and groovy from moqui
>
> <if condition="settlementTerm?.orderPmtServiceRegisterId">
> <then>
> <service-call name="update#mantle.order.OrderPart"
> in-map="[orderId:orderId,
> orderPartSeqId:orderPartSeqId, settlementTermId:settlementTermId]"/>
>
> <entity-find-one entity-name="moqui.service.ServiceRegister"
> value-field="serviceRegister">
> <field-map field-name="serviceRegisterId"
> from="settlementTerm.orderPmtServiceRegisterId"/>
> </entity-find-one>
> <script><![CDATA[
> try {
> ec.service.sync().name(serviceRegister.serviceName).parameters(context).softValidate(true).disableAuthz().call()
> if (ec.message.hasError()) return
> } catch (Throwable t) {
> ec.logger.log(300, "Error calling auto order payments
> service for SettlementTerm ${settlementTermId}", t)
> }
> ]]></script>
> </then>
> <else>
> <service-call
> name="mantle.account.PaymentServices.create#Payment" out-map="context"
> in-map="context + [amount:amount,
> amountUomId:amountUomId, fromPartyId:fromPartyId, toPartyId:toPartyId]"/>
> </else>
> </if>
>
> [2] https://ci.apache.org/projects/ofbiz/site/trunk/javadocs/
>