Re: Important Changes to Trunk and Use of Ant & Gradle

[Taher Alkhateeb <sl...@gmail.com>]

Hello Everyone,

I am very happy and thrilled to announce that OFBiz now has a skeleton for
a plug-in system without any changes to the code base :)

How did we design it
------------------------------
- Add all the components defined in framework/component-load.xml as gradle
subprojects
- Add all the components defined in applications/component-load.xml as
gradle subprojects
- Add all the components defined in specialcomponents/component-load.xml as
gradle subprojects
- Add the start component as a gradle subproject
- Add all the components in /hot-deploy as gradle subprojects
- Each subproject can _optionally_ have a build.gradle script
- If a directory exists in framework, applications or specialpurpose but is
not defined in component-load.xml, then it will be considered a dead
folder, it will not even compile and it will not be part of OFBiz and your
IDE will ignore it.

What are the benefits
-------------------------------
- Each subproject can declare its own library dependencies in its own
build.gradle away from the framework
- Compilation, running, and testing is centralized at the master build
script, making the subproject build scripts very small
- It is very easy now to "turn on" or "turn off" a plugin with a build
script that simply removes/adds it to component-load.xml!
- We can now design a full plug-in API and can design a full eco-system
around OFBiz.
- If we decide to, we do not _have to_ compile and test everything, we can
offload the maintenance of plugins to specialized teams, or even to third
parties.

So, maximum benefits without any code changes (except for may 30-40 lines
in the build scripts). I have a few ideas for a full API design in the
future (another thread perhaps) for these plugins (in gradle). The API
could contain tasks like:
- installPlugin
- activatePlugin
- deactivatePlugin
- uninstallPlugin
- etc ...

The latest patch in https://issues.apache.org/jira/browse/OFBIZ-7534 has
everything that I described above with all tests passing. Try it and enjoy.
Just comment-out any component in component-load.xml and observe how it
does not exist anymore :)



On Mon, Jun 27, 2016 at 6:30 PM, Taher Alkhateeb <slidingfilaments@gmail.com
> wrote:

> Hey Everyone,
>
> I am exhausted, but I have a working patch in
> https://issues.apache.org/jira/browse/OFBIZ-7534
>
> This patch introduces gradle with all tests passing. I have also added a
> new README.md file which documents how to use this system. I think you will
> find the README.md file valuable for guidance on how to use and test. This
> is a big patch and requires a LOT of testing, so I need everyone to jump on
> board and help please.
>
> Oh, and I have a little rant
>
> ====== rant begins ======
> The classpath issues I faced with OFBiz are really horrible, and the
> design of the code base around classpaths is one of the worst I've even
> seen in my life. I cannot even fathom why we seperate classpaths for
> components and use this weird custom classloader. It took me so much time
> and thinking in delivering this patch just because of this damn issue
> ====== rant ends ======
>
> The executable is huge if you type ps ax|grep java. This is because of the
> unfortunately large and complex classpath. I will clean that up slowly as
> we progress.
>
> I look forward to your help and feedback.
>
> Taher Alkhateeb
>
> On Thu, Jun 23, 2016 at 6:23 PM, Taher Alkhateeb <
> slidingfilaments@gmail.com> wrote:
>
>> Hey Guys,
>>
>> I've attached a text file in
>> https://issues.apache.org/jira/browse/OFBIZ-7534
>>
>> This text file shows the old ant commands and how I implemented them
>> (most of them anyway) in Gradle. Please review it and provide your feedback:
>>
>> - Does it look correct?
>> - Are the parameters corrects?
>> - Are there any important missing tasks?
>> - Do you want to provide shorter versions for some of the tasks?
>>
>> Note, I only created two real short commands for OFBiz server commands:
>> loadDemo and testInteg. These exist because the most common command in
>> OFBiz would be ./gradlew cleanAll loadDemo testInteg
>>
>> I look forward to your feedback.
>>
>> Taher Alkhateeb
>>
>> On Wed, Jun 22, 2016 at 11:12 PM, Taher Alkhateeb <
>> slidingfilaments@gmail.com> wrote:
>>
>>> Hi Jacques,
>>>
>>> Thank you for the explanation. OK, between your feedback and the votes
>>> from other members, I'll keep them. Working on integration in gradle ...
>>>
>>> Cheers
>>>
>>> Taher Alkhateeb
>>>
>>> On Wed, Jun 22, 2016 at 11:08 PM, Jacques Le Roux <
>>> jacques.le.roux@les7arts.com> wrote:
>>>
>>>> Hi Taher,
>>>>
>>>> Inline...
>>>>
>>>> Le 22/06/2016 à 16:23, Taher Alkhateeb a écrit :
>>>>
>>>>> Hi Michael Jacques and everyone,
>>>>>
>>>>> So I also want to confirm that you really need the following tasks:
>>>>>
>>>>> - build-dev
>>>>> - build-production
>>>>> - build-qa
>>>>> - build-test
>>>>> - revert-dev
>>>>>
>>>>
>>>> I introduced those for a custom project and improved it in the
>>>> following project I worked on. So yes I use these targets. BTW I documented
>>>> it at
>>>>
>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Addressing+Custom+Requirements+In+OFBiz#AddressingCustomRequirementsInOFBiz-Therearealso4deploymenttargetsavailableinthemainbuild.xml
>>>> .
>>>>
>>>> Are you using them yourselves? Here are the reasons why I suggest to
>>>>> remove
>>>>> them:
>>>>> - First, you must have the patch command existing in your environment,
>>>>> thus
>>>>> making the build script brittle
>>>>>
>>>>
>>>> Actually it's a bit more complicated than that, see the comment above
>>>> the calculate-svn-patch-available target:
>>>> <!-- Following allow to use "svn patch" and fallback on "patch" if
>>>> necessary -->
>>>> and then
>>>> <!-- On Linux prefer patch because "svn patch" needs "ant exec and you
>>>> can't check patching errors -->
>>>>
>>>> - Second, it has this weird naming convention for patches and how they
>>>>> should be applied. I would imagine that every team and every company
>>>>> has
>>>>> its own style and methodology for patching and keeping track of files
>>>>> and
>>>>> changes. They use their own version control system and even different
>>>>> kinds
>>>>> of patches
>>>>> - It is dependent on a specific format of diff which you must generate
>>>>> either with svn diff or diff -Naur.
>>>>>
>>>>
>>>> This is useful even if you are using Git for your project. Why? Because
>>>> if, instead of relying on a released package, you decide to rely on the
>>>> trunk or a release branch you need to connect your project to the ASF svn
>>>> repo.
>>>> And yes it's possible to have both, I did it, used those targets and
>>>> was quite happy with them. I can agree it's a bit complicated, but not that
>>>> much when you look into details: they make sense.
>>>>
>>>> It seems like it does not make a lot of sense to keep something like
>>>>> that
>>>>> when companies might completely ignore it and choose their own
>>>>> implementation strategies. Do you agree? Feedback?
>>>>>
>>>>
>>>> You got mine :)
>>>>
>>>> Jacques
>>>>
>>>>
>>>>
>>>>> Taher Alkhateeb
>>>>>
>>>>> On Wed, Jun 22, 2016 at 4:01 PM, Jacques Le Roux <
>>>>> jacques.le.roux@les7arts.com> wrote:
>>>>>
>>>>> I wonder though why this was introduced. If we have no feedback from
>>>>>> any
>>>>>> member of the community I think we can drop it. I don't see how to
>>>>>> use it.
>>>>>>
>>>>>> Jacques
>>>>>>
>>>>>>
>>>>>>
>>>>>> Le 22/06/2016 à 14:43, Michael Brohl a écrit :
>>>>>>
>>>>>> Hi Taher,
>>>>>>>
>>>>>>> no, I don't use it myself. This was just a guess by looking what the
>>>>>>> task
>>>>>>> does and it seemed to me this should be preserved.
>>>>>>> No problem to drop it from my side.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Michael Brohl
>>>>>>> ecomify GmbH
>>>>>>> www.ecomify.de
>>>>>>>
>>>>>>>
>>>>>>> Am 22.06.16 um 14:37 schrieb Taher Alkhateeb:
>>>>>>>
>>>>>>> Hi Michael,
>>>>>>>>
>>>>>>>> Are you sure you are using "run-test-list"? This is an old task
>>>>>>>> using a
>>>>>>>> target on the server called --testlist which essentially creates an
>>>>>>>> ant
>>>>>>>> file of all the suites in OFBiz and runs them one-by-one stopping
>>>>>>>> OFBiz
>>>>>>>> in
>>>>>>>> between each run.
>>>>>>>>
>>>>>>>> To me it seems like run-tests is doing the same thing but faster. Is
>>>>>>>> this a
>>>>>>>> typo by you or are you actually using this target? The reason I ask
>>>>>>>> is
>>>>>>>> because I actually wanted to delete the entire
>>>>>>>> TestListContainer.java and
>>>>>>>> related files in the future. But now of course I hesitate.
>>>>>>>>
>>>>>>>> Thanks in advance for your feedback.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Taher Alkhateeb
>>>>>>>>
>>>>>>>> On Wed, Jun 22, 2016 at 12:00 AM, Michael Brohl <
>>>>>>>> michael.brohl@ecomify.de>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Taher,
>>>>>>>>
>>>>>>>>> First question:
>>>>>>>>>
>>>>>>>>> Don't drop:
>>>>>>>>>
>>>>>>>>> - build-dev
>>>>>>>>> - build-production
>>>>>>>>> - build-qa
>>>>>>>>> - build-test
>>>>>>>>> - refresh
>>>>>>>>> - revert-dev
>>>>>>>>> - run-test-list
>>>>>>>>>
>>>>>>>>> Drop:
>>>>>>>>>
>>>>>>>>> - clean-ivy (assuming this is not needed if we use Gradle)
>>>>>>>>> - copy-dtds
>>>>>>>>> - download-PG-JDBC
>>>>>>>>> - download-activemq
>>>>>>>>> - download-mySQL-JDBC (download tasks: assuming they are not
>>>>>>>>> needed if
>>>>>>>>> we
>>>>>>>>> use Gradle/automatic dependency mechanism)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Not sure (I don't use them):
>>>>>>>>>
>>>>>>>>> - create-tenant
>>>>>>>>> - load-all-tenants
>>>>>>>>> - load-demo-multitenant
>>>>>>>>> - load-tenant
>>>>>>>>> - load-tenant-data-readers
>>>>>>>>> - gen-kek
>>>>>>>>> - gitinfo
>>>>>>>>> - run-test-list
>>>>>>>>> - start-batch-secure
>>>>>>>>> - start-both-secure
>>>>>>>>> - start-pos-secure
>>>>>>>>> - start-secure (...secure: not sure if these are needed anymore, I
>>>>>>>>> think
>>>>>>>>> Jacques can give some hints about them)
>>>>>>>>> - svninfo
>>>>>>>>>
>>>>>>>>> Second question: +1
>>>>>>>>>
>>>>>>>>> but I'm not sure about the load-tenant... tasks (I don't use them).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Michael Brohl
>>>>>>>>> ecomify GmbH
>>>>>>>>> www.ecomify.de
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Am 21.06.16 um 22:09 schrieb Taher Alkhateeb:
>>>>>>>>>
>>>>>>>>> Hi Everyone,
>>>>>>>>>
>>>>>>>>>> I have create the JIRA
>>>>>>>>>> https://issues.apache.org/jira/browse/OFBIZ-7534
>>>>>>>>>> for
>>>>>>>>>> this project
>>>>>>>>>>
>>>>>>>>>> I have two questions in this thread
>>>>>>>>>>
>>>>>>>>>> First Question
>>>>>>>>>> ------------------
>>>>>>>>>> Can I drop the below tasks from the build system? They currently
>>>>>>>>>> exist
>>>>>>>>>> in
>>>>>>>>>> Ant but I am not sure whether they are actively used or not. So
>>>>>>>>>> if you
>>>>>>>>>> want
>>>>>>>>>> me to add any of these tasks please reply to this thread,
>>>>>>>>>> otherwise I
>>>>>>>>>> will
>>>>>>>>>> not include them in gradle. Please note I added all multi tenant
>>>>>>>>>> tasks
>>>>>>>>>> because many of them are broken or have no functionality, so I am
>>>>>>>>>> assuming
>>>>>>>>>> people are doing multi-tenancy manually but not sure, so please
>>>>>>>>>> help!
>>>>>>>>>>
>>>>>>>>>> - build-dev
>>>>>>>>>> - build-production
>>>>>>>>>> - build-qa
>>>>>>>>>> - build-test
>>>>>>>>>> - clean-ivy
>>>>>>>>>> - copy-dtds
>>>>>>>>>> - create-tenant
>>>>>>>>>> - download-PG-JDBC
>>>>>>>>>> - download-activemq
>>>>>>>>>> - download-mySQL-JDBC
>>>>>>>>>> - gen-kek
>>>>>>>>>> - gitinfo
>>>>>>>>>> - load-all-tenants
>>>>>>>>>> - load-demo-multitenant
>>>>>>>>>> - load-tenant
>>>>>>>>>> - load-tenant-data-readers
>>>>>>>>>> - refresh
>>>>>>>>>> - revert-dev
>>>>>>>>>> - run-test-list
>>>>>>>>>> - start-batch
>>>>>>>>>> - start-batch-secure
>>>>>>>>>> - start-both-secure
>>>>>>>>>> - start-pos-secure
>>>>>>>>>> - start-secure
>>>>>>>>>> - svninfo
>>>>>>>>>>
>>>>>>>>>> Second Question
>>>>>>>>>> -----------------------
>>>>>>>>>>
>>>>>>>>>> it seems many of the load tasks are too specific. So I suggest to
>>>>>>>>>> only
>>>>>>>>>> implement loadDemo and the rest are executed manually by users,
>>>>>>>>>> for
>>>>>>>>>> example: ./gradlew 'ofbiz --load-data reader=seed, seed-initial,
>>>>>>>>>> ext'
>>>>>>>>>> instead of load-extseed.
>>>>>>>>>>
>>>>>>>>>> If you would like to add the other load data tasks, please specify
>>>>>>>>>> which
>>>>>>>>>> ones.
>>>>>>>>>>
>>>>>>>>>> Appreciate your early responses.
>>>>>>>>>>
>>>>>>>>>> Taher Alkhateeb
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>
>>>
>>
>