You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openjpa.apache.org by Mark Struberg <st...@yahoo.de.INVALID> on 2023/05/25 16:36:04 UTC
enhance without Serp
Hi!
Right now I'm trying to get rid of serp step by step.
The code is right now at my own github repo in the fb_asmEnhance branch:
https://github.com/struberg/openjpa/tree/fb_asmEnhance
The approach I took for now is to create a few methods in AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other way around. That way we should be able to replace functionality part by part but still keep all things afloat.
For now I've started with the PCSubclassValidator.
Right now this evaluates the attributs using Serp plus ASM and then compare the results.
If something is fishy, you'll see the following
throw new IllegalStateException("MSX ASMTODO " + bcField + " " + field);
It would be extremely helpful, if you could go through your projects and let it run and report how it's going.
If you see that "MSX ASMTODO" somewhere then we know I messed something up.
I'll gonna finally remove the BCClass handling from those parts in a few days. Current commit is
https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
txs and LieGrue,
strub
Re: enhance without Serp - Update
Posted by Francesco Chicchiriccò <il...@apache.org>.
Or maybe the issue is just the enhancement order?
In the JPARealm sample below, in fact, I can see that JPARealm.class is not yet enhanced so the error is raised while enhancing, and possibly because JPAAccountPolicy was not enhanced yet.
On 03/07/23 17:00, Francesco Chicchiriccò wrote:
> Adding some info.
>
> First of all, the class to which the error message is referring is not always the same.
>
> Secondly, at least about AccountPolicy as reported below, the original error message is stating
>
> Execution enhancer of goal org.apache.openjpa:openjpa-maven-plugin:4.0.0-SNAPSHOT:enhance failed: An error occurred while enhancing org.apache.syncope.core.persistence.jpa.entity.JPARealm. Exception message: Type org/apache/syncope/core/persistence/api/entity/policy/AccountPolicy not present: org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
>
> If you look at JPARealm:
>
> https://github.com/apache/syncope/blob/master/core/persistence-jpa/src/main/java/org/apache/syncope/core/persistence/jpa/entity/JPARealm.java#L82
>
> it declares
>
> @ManyToOne(fetch = FetchType.EAGER)
> private JPAAccountPolicy accountPolicy;
>
> e.g. a reference to the JPA entity JPAAccountPolicy, not to the interface that this class implements, e.g. AccountPolicy
>
> It looks to me that somewhere the OpenJPA enhancement process is trying to resolve more than it should do.
>
> Does this make ringing any bell?
>
> Regards.
>
> On 03/07/23 16:32, Francesco Chicchiriccò wrote:
>> On 03/07/23 16:27, Mark Struberg wrote:
>>> Went forward and compiled the Syncope main branch.
>>>
>>> Getting this error when running with mvn -X
>>>
>>> Caused by: java.lang.ClassNotFoundException: org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
>>> at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass (SelfFirstStrategy.java:50)
>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass (ClassRealm.java:271)
>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:247)
>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:239)
>>> at java.lang.Class.forName0 (Native Method)
>>> at java.lang.Class.forName (Class.java:467)
>>> at org.apache.xbean.asm9.ClassWriter.getCommonSuperClass (ClassWriter.java:1043)
>>> at org.apache.xbean.asm9.SymbolTable.addMergedType (SymbolTable.java:1202)
>>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1299)
>>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1244)
>>> at org.apache.xbean.asm9.MethodWriter.computeAllFrames (MethodWriter.java:1611)
>>> at org.apache.xbean.asm9.MethodWriter.visitMaxs (MethodWriter.java:1547)
>>> at org.apache.xbean.asm9.tree.MethodNode.accept (MethodNode.java:767)
>>> at org.apache.xbean.asm9.tree.MethodNode.accept (MethodNode.java:647)
>>> at org.apache.xbean.asm9.tree.ClassNode.accept (ClassNode.java:468)
>>> at org.apache.openjpa.util.asm.AsmHelper.readIntoBCClass (AsmHelper.java:146)
>>> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:606)
>>> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:5889)
>>> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:5829)
>>> at org.apache.openjpa.enhance.PCEnhancer$1.run (PCEnhancer.java:5799)
>>> at org.apache.openjpa.lib.conf.Configurations.launchRunnable (Configurations.java:760)
>>> at org.apache.openjpa.lib.conf.Configurations.runAgainstAllAnchors (Configurations.java:745)
>>>
>>>
>>> Is this the one you experience too?
>>
>> Yes, exactly.
>>
>> As you might have found out, AccountPolicy is an interface implemented by JPAAccountPolicy, which is the corresponding JPA entity.
>> We take this approach for all JPA entities, actually.
>>
>> I thought the reason why AccountPolicy or JPAAccountPolicy are not found is that enhancement does not actually finds them.
>>
>>> Will take a look at it now.
>>
>> Thank you very much!
>>
>> Regards.
>>>> Am 03.07.2023 um 15:49 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>>>
>>>> Hi Francesco!
>>>>
>>>> Can you see which one of the classes fail to enhance?
>>>>
>>>> In which case you might try to run the enhancement of that very class manually in your IDE.
>>>>
>>>> Main class: org.apache.openjpa.enhance.PCEnhancer
>>>> VMOptions:: -ea -Dopenjpa.ConnectionURL=jdbc:derby:target/database/openjpa-derby-database;create=true -Dopenjpa.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver
>>>> or something similar for your connection
>>>>
>>>> Then you also need the parameter pointing to the class file which you want to enhance, e.g.
>>>> target/test-classes/org/apache/openjpa/persistence/proxy/entities/Annuity.class
>>>>
>>>> Please keep me updated!
>>>>
>>>>
>>>> LieGrue,
>>>> strub
>>>>
>>>> PS: does it add value if we'd write all this up? You will likely know most of that stuff, but others who dig into OpenJPA do not.
>>>>
>>>>
>>>>
>>>>> Am 03.07.2023 um 15:28 schrieb Francesco Chicchiriccò <il...@apache.org>:
>>>>>
>>>>> Hi Mark,
>>>>> thanks for your reply.
>>>>>
>>>>> So I am proceeding as follows.
>>>>>
>>>>> Took the executions at
>>>>>
>>>>> https://ci-builds.apache.org/job/OpenJPA/job/OpenJPA-master-deploy/
>>>>>
>>>>> (the Jenkins job which is publishing SNAPSHOT artifacts from master to ASF Maven repo) and saw that the artifacts with which the Syncope build succeeded last Friday are the one from May 16th, so the corresponding commit id is
>>>>>
>>>>> b238417dabeac935846fe1ce2fef28aafaeef205
>>>>>
>>>>> I confirm that building locally from that commit id everything is working.
>>>>>
>>>>> As soon as I move to next commit (dated May 22nd, pushed and deployed today) with id
>>>>>
>>>>> b51d003ed98aa7663465aad7c9a1b554b006cc7d
>>>>>
>>>>> I see the same behavior as with HEAD, e.g.
>>>>>
>>>>> 2b9b024f273d63e479a02cad751c28b8ef974ace
>>>>>
>>>>> I will try to take a closer look at the code.
>>>>> Regards.
>>>>>
>>>>> On 03/07/23 14:44, Mark Struberg wrote:
>>>>>> Hi Francesco!
>>>>>>
>>>>>> That's very valuable information!
>>>>>> Could you please do a git-bisect to find out which change did break it?
>>>>>>
>>>>>> Actually the algorithm to select which classes are to be enhanced should not have changed. Thus I'm really curious what I did break.
>>>>>>
>>>>>> Most times the amount of enhanced classes are the same but slightly differ in the generated code. That might be due to now we really generate Java11++ code whereas before we only generated Java 1.1 code (yes, not even 1.5!)
>>>>>>
>>>>>> What I usually do is to have 2 checkouts. You might want to change the pom GAV version of the working checkout to something something different so you might have both in your ~/.m2/repository in parallel.
>>>>>>
>>>>>> The first thing I do is to compare the class files in target/classes of both versions with 'compare with clipboard'.
>>>>>> I just open the class file in target/classes and let Intellij/Netbeans do the decompilation for me.
>>>>>>
>>>>>> You might find some differences here in which case this needs further investigation. Note that the order of the methods might have changed.
>>>>>> Sometimes you will also see some 'cannot decompile' in the code block. This happens when something in the bytecode is messed up. You can also watch out for ClassNotFound or VerifyError, etc.
>>>>>>
>>>>>> If this is the case I then use
>>>>>> $> javap -c -p -constants target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile
>>>>>> on both versions and check whether something is different.
>>>>>> Often it's a wrong offset in the stack calculation (e.g. ALOAD 1 vs ALOAD 2)
>>>>>>
>>>>>> Note that the 'old' Java 1.1 bytecode doesn't understand LDC of Class constants (LoaD Constant). This was only introduced in Java 1.4 or 1.5 bytecode afair. Before that a dynamic Class.forName was called and stored in a static variable. Ugly stuff, and happy to finally get rid of all those hacks...
>>>>>>
>>>>>> Hope that helps to get you started. Feel free to ping me again over here or at the #apachee ASF slack channel.
>>>>>>
>>>>>> txs and LieGrue,
>>>>>> strub
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò <il...@apache.org>:
>>>>>>>
>>>>>>> From a very preliminary analysis it seems that just a subset of the persistence-capable classes are effectively found and enhanced: now [2], it used to be [3].
>>>>>>>
>>>>>>> I'll continue investigating.
>>>>>>>
>>>>>>> Regards.
>>>>>>>
>>>>>>> On 03/07/23 13:56, Francesco Chicchiriccò wrote:
>>>>>>>> Hi Mark,
>>>>>>>> the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT is now failing - it used to work at least until last June 30th.
>>>>>>>>
>>>>>>>> The issues seem to be related to entity enhancement, which we perform via openjpa-maven-plugin - see [1].
>>>>>>>>
>>>>>>>> Is there anything obvious we should be changing?
>>>>>>>> Meanwhile I'll try to dig to see if I am able to understand something more.
>>>>>>>>
>>>>>>>> Regards.
>>>>>>>>
>>>>>>>> [1] https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174
>>>>>>> [2] https://paste.apache.org/gxcm6
>>>>>>> [3] https://paste.apache.org/1g9k5
>>>>>>>
>>>>>>>> On 03/07/23 12:15, Mark Struberg wrote:
>>>>>>>>> Update: I've now merged the wip to our master and pushed it to the ASF gitox repo.
>>>>>>>>>
>>>>>>>>> I think we already came pretty far and it looks really promising. So there is a high chance that we succeed, although there is still plenty of work in front of us.
>>>>>>>>>
>>>>>>>>> How it works:
>>>>>>>>>
>>>>>>>>> Basically there are 2 sets of information in the PCEnhancer right now:
>>>>>>>>>
>>>>>>>>> _managed: the BCClass version of the original class
>>>>>>>>> _pc: the BCClass of the generated/modified class. Might be the same as _managed but in case of subclassing or interface case it is different. In those cases _managed represents the original class and _pc the subclass/concrete implementation.
>>>>>>>>>
>>>>>>>>> managed and pc are the ASM variants of _managed and _pc.
>>>>>>>>>
>>>>>>>>> The main entry point for debugging into it is PCEnhancer#run()
>>>>>>>>>
>>>>>>>>> There are a few methods in ASMHelper to update from BCClass -> ASM and vice versa:
>>>>>>>>>
>>>>>>>>> AsmHelper.readIntoBCClass(pc, _pc);
>>>>>>>>> takes all the information from the ASM pc, creates a class byte[] and reads that back into the existing _pc instance.
>>>>>>>>>
>>>>>>>>> pc = AsmHelper.toClassNode(_pc);
>>>>>>>>> is the corresponding method in the other direction. from BCClass to ASM.
>>>>>>>>>
>>>>>>>>> Right now we have the following methods left to migrate in PCEnhancer
>>>>>>>>> addAttachDetachCode();
>>>>>>>>> addSerializationCode();
>>>>>>>>> addCloningCode();
>>>>>>>>> runAuxiliaryEnhancers();
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'll be off for a week on holidays in Venice.
>>>>>>>>> If someone want's to join in with the effort then please go on! You might want to compare the current PCEnhancer code with an older version to get an idea how it can be done and to see some patterns I've used.
>>>>>>>>>
>>>>>>>>> As a quick hint: Java is basically a stack based virtual CPU. The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that comes the parameters. The size of the param on the stack depends on the type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
>>>>>>>>> In case of a static method there is ofc no 'this' on the stack, so the parameters start with zero offset.
>>>>>>>>> After all the parameters there are the unnamed 'local' variables. I tried to consistently name them xxxVarPos.
>>>>>>>>> See also the various helper methods in ASMHelper. For example for the various LOAD, RETURN, STORE operations which depends based on the type (ALOAD vs ILOAD vs IALOAD etc)
>>>>>>>>> Also read the guide to ASM PDF from objectweb.
>>>>>>>>>
>>>>>>>>> I'll be mostly offline till Sunday, but will try to answer mails if there are some questions.
>>>>>>>>>
>>>>>>>>> txs and LieGrue,
>>>>>>>>> strub
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <ma...@gmail.com>:
>>>>>>>>>>
>>>>>>>>>> +1 thanks for tackling this, Mark!
>>>>>>>>>>
>>>>>>>>>> ASM definitely more widely used going forward.
>>>>>>>>>>
>>>>>>>>>> -Matt
>>>>>>>>>>
>>>>>>>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg <st...@yahoo.de.INVALID> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Small update:
>>>>>>>>>>>
>>>>>>>>>>> I've worked on it over the last few weeks, and I'm getting closer
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>>>>
>>>>>>>>>>> contains the latest work on the PCEnhancer. Right in the middle of replacing Serp with native ASM code.
>>>>>>>>>>>
>>>>>>>>>>> Note that I also had to modify a few interfaces with a few more to follow from BCClass to ASM ClassNode.
>>>>>>>>>>>
>>>>>>>>>>> Happy to get some feedback!
>>>>>>>>>>>
>>>>>>>>>>> If anybody wants to help with this effort I'm happy to also push this feature branch to our ASF repo. It looks reasonably promising already.
>>>>>>>>>>>
>>>>>>>>>>> LieGrue,
>>>>>>>>>>> strub
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi!
>>>>>>>>>>>>
>>>>>>>>>>>> Right now I'm trying to get rid of serp step by step.
>>>>>>>>>>>> The code is right now at my own github repo in the fb_asmEnhance branch:
>>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The approach I took for now is to create a few methods in AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other way around. That way we should be able to replace functionality part by part but still keep all things afloat.
>>>>>>>>>>>>
>>>>>>>>>>>> For now I've started with the PCSubclassValidator.
>>>>>>>>>>>> Right now this evaluates the attributs using Serp plus ASM and then compare the results.
>>>>>>>>>>>> If something is fishy, you'll see the following
>>>>>>>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + " " + field);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It would be extremely helpful, if you could go through your projects and let it run and report how it's going.
>>>>>>>>>>>> If you see that "MSX ASMTODO" somewhere then we know I messed something up.
>>>>>>>>>>>>
>>>>>>>>>>>> I'll gonna finally remove the BCClass handling from those parts in a few days. Current commit is
>>>>>>>>>>>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>>>>>>>>>>>>
>>>>>>>>>>>> txs and LieGrue,
>>>>>>>>>>>> strub
>>
>
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Re: enhance without Serp - Update
Posted by Francesco Chicchiriccò <il...@apache.org>.
Thanks Mark, your last commit seems to have fixed, our build is green again.
Nice job!
Il lun 3 lug 2023, 18:10 Mark Struberg <st...@yahoo.de.invalid> ha
scritto:
> Francesco, I've now pushed this change. Can you please try it out? Looks
> good over here now.
> Now I started the full Syncope build and hope all comes green again.
>
> txs and LieGrue,
> strub
>
> > Am 03.07.2023 um 17:55 schrieb Mark Struberg <st...@yahoo.de>:
> >
> > Think I found the solution, but first want to run all tests in OpenJPA.
> >
> > I did use the standard ClassWriter in ASMHelper. This does use
> Class.forName.
> > Changed ASMHelper to use the BCClassWriter you wrote some time ago. That
> should do the trick I hope.
> >
> > Gimme 10 minutes plz.
> >
> > LieGrue,
> > strub
> >
> >
> >> Am 03.07.2023 um 17:34 schrieb Romain Manni-Bucau <
> rmannibucau@gmail.com>:
> >>
> >> @Francesco: can you *javap -v -cp $yourclasspath the.class.which.Fail*
> when
> >> it works vs when it fails please?
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> >> <https://rmannibucau.metawerx.net/> | Old Blog
> >> <http://rmannibucau.wordpress.com> | Github <
> https://github.com/rmannibucau> |
> >> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> >> <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> >>
> >>
> >> Le lun. 3 juil. 2023 à 17:23, Mark Struberg <st...@yahoo.de.invalid>
> a
> >> écrit :
> >>
> >>> Does not yet ring any bells, sorry :(
> >>>
> >>> Right now trying to fire up the enhancer manually in the debugger.
> >>> Difference really is that we now do the ASM part way earlier, but still
> >>> class per class.
> >>> Before that it was done as part of the ASMAdapter.
> >>>
> >>> Still digging.
> >>>
> >>> LieGrue,
> >>> strub
> >>>
> >>>
> >>>> Am 03.07.2023 um 17:00 schrieb Francesco Chicchiriccò <
> >>> ilgrosso@apache.org>:
> >>>>
> >>>> Adding some info.
> >>>>
> >>>> First of all, the class to which the error message is referring is not
> >>> always the same.
> >>>>
> >>>> Secondly, at least about AccountPolicy as reported below, the original
> >>> error message is stating
> >>>>
> >>>> Execution enhancer of goal
> >>> org.apache.openjpa:openjpa-maven-plugin:4.0.0-SNAPSHOT:enhance failed:
> An
> >>> error occurred while enhancing
> >>> org.apache.syncope.core.persistence.jpa.entity.JPARealm. Exception
> message:
> >>> Type
> org/apache/syncope/core/persistence/api/entity/policy/AccountPolicy
> >>> not present:
> >>> org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
> >>>>
> >>>> If you look at JPARealm:
> >>>>
> >>>>
> >>>
> https://github.com/apache/syncope/blob/master/core/persistence-jpa/src/main/java/org/apache/syncope/core/persistence/jpa/entity/JPARealm.java#L82
> >>>>
> >>>> it declares
> >>>>
> >>>> @ManyToOne(fetch = FetchType.EAGER)
> >>>> private JPAAccountPolicy accountPolicy;
> >>>>
> >>>> e.g. a reference to the JPA entity JPAAccountPolicy, not to the
> >>> interface that this class implements, e.g. AccountPolicy
> >>>>
> >>>> It looks to me that somewhere the OpenJPA enhancement process is
> trying
> >>> to resolve more than it should do.
> >>>>
> >>>> Does this make ringing any bell?
> >>>>
> >>>> Regards.
> >>>>
> >>>> On 03/07/23 16:32, Francesco Chicchiriccò wrote:
> >>>>> On 03/07/23 16:27, Mark Struberg wrote:
> >>>>>> Went forward and compiled the Syncope main branch.
> >>>>>>
> >>>>>> Getting this error when running with mvn -X
> >>>>>>
> >>>>>> Caused by: java.lang.ClassNotFoundException:
> >>> org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
> >>>>>> at
> >>> org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass
> >>> (SelfFirstStrategy.java:50)
> >>>>>> at
> >>>
> org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass
> >>> (ClassRealm.java:271)
> >>>>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass
> >>> (ClassRealm.java:247)
> >>>>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass
> >>> (ClassRealm.java:239)
> >>>>>> at java.lang.Class.forName0 (Native Method)
> >>>>>> at java.lang.Class.forName (Class.java:467)
> >>>>>> at org.apache.xbean.asm9.ClassWriter.getCommonSuperClass
> >>> (ClassWriter.java:1043)
> >>>>>> at org.apache.xbean.asm9.SymbolTable.addMergedType
> >>> (SymbolTable.java:1202)
> >>>>>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1299)
> >>>>>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1244)
> >>>>>> at org.apache.xbean.asm9.MethodWriter.computeAllFrames
> >>> (MethodWriter.java:1611)
> >>>>>> at org.apache.xbean.asm9.MethodWriter.visitMaxs
> >>> (MethodWriter.java:1547)
> >>>>>> at org.apache.xbean.asm9.tree.MethodNode.accept
> >>> (MethodNode.java:767)
> >>>>>> at org.apache.xbean.asm9.tree.MethodNode.accept
> >>> (MethodNode.java:647)
> >>>>>> at org.apache.xbean.asm9.tree.ClassNode.accept
> >>> (ClassNode.java:468)
> >>>>>> at org.apache.openjpa.util.asm.AsmHelper.readIntoBCClass
> >>> (AsmHelper.java:146)
> >>>>>> at org.apache.openjpa.enhance.PCEnhancer.run
> (PCEnhancer.java:606)
> >>>>>> at org.apache.openjpa.enhance.PCEnhancer.run
> >>> (PCEnhancer.java:5889)
> >>>>>> at org.apache.openjpa.enhance.PCEnhancer.run
> >>> (PCEnhancer.java:5829)
> >>>>>> at org.apache.openjpa.enhance.PCEnhancer$1.run
> >>> (PCEnhancer.java:5799)
> >>>>>> at org.apache.openjpa.lib.conf.Configurations.launchRunnable
> >>> (Configurations.java:760)
> >>>>>> at
> >>> org.apache.openjpa.lib.conf.Configurations.runAgainstAllAnchors
> >>> (Configurations.java:745)
> >>>>>>
> >>>>>>
> >>>>>> Is this the one you experience too?
> >>>>>
> >>>>> Yes, exactly.
> >>>>>
> >>>>> As you might have found out, AccountPolicy is an interface
> implemented
> >>> by JPAAccountPolicy, which is the corresponding JPA entity.
> >>>>> We take this approach for all JPA entities, actually.
> >>>>>
> >>>>> I thought the reason why AccountPolicy or JPAAccountPolicy are not
> >>> found is that enhancement does not actually finds them.
> >>>>>
> >>>>>> Will take a look at it now.
> >>>>>
> >>>>> Thank you very much!
> >>>>>
> >>>>> Regards.
> >>>>>>> Am 03.07.2023 um 15:49 schrieb Mark Struberg
> >>> <st...@yahoo.de.INVALID>:
> >>>>>>>
> >>>>>>> Hi Francesco!
> >>>>>>>
> >>>>>>> Can you see which one of the classes fail to enhance?
> >>>>>>>
> >>>>>>> In which case you might try to run the enhancement of that very
> class
> >>> manually in your IDE.
> >>>>>>>
> >>>>>>> Main class: org.apache.openjpa.enhance.PCEnhancer
> >>>>>>> VMOptions:: -ea
> >>>
> -Dopenjpa.ConnectionURL=jdbc:derby:target/database/openjpa-derby-database;create=true
> >>> -Dopenjpa.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver
> >>>>>>> or something similar for your connection
> >>>>>>>
> >>>>>>> Then you also need the parameter pointing to the class file which
> you
> >>> want to enhance, e.g.
> >>>>>>>
> >>>
> target/test-classes/org/apache/openjpa/persistence/proxy/entities/Annuity.class
> >>>>>>>
> >>>>>>> Please keep me updated!
> >>>>>>>
> >>>>>>>
> >>>>>>> LieGrue,
> >>>>>>> strub
> >>>>>>>
> >>>>>>> PS: does it add value if we'd write all this up? You will likely
> know
> >>> most of that stuff, but others who dig into OpenJPA do not.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Am 03.07.2023 um 15:28 schrieb Francesco Chicchiriccò <
> >>> ilgrosso@apache.org>:
> >>>>>>>>
> >>>>>>>> Hi Mark,
> >>>>>>>> thanks for your reply.
> >>>>>>>>
> >>>>>>>> So I am proceeding as follows.
> >>>>>>>>
> >>>>>>>> Took the executions at
> >>>>>>>>
> >>>>>>>>
> https://ci-builds.apache.org/job/OpenJPA/job/OpenJPA-master-deploy/
> >>>>>>>>
> >>>>>>>> (the Jenkins job which is publishing SNAPSHOT artifacts from
> master
> >>> to ASF Maven repo) and saw that the artifacts with which the Syncope
> build
> >>> succeeded last Friday are the one from May 16th, so the corresponding
> >>> commit id is
> >>>>>>>>
> >>>>>>>> b238417dabeac935846fe1ce2fef28aafaeef205
> >>>>>>>>
> >>>>>>>> I confirm that building locally from that commit id everything is
> >>> working.
> >>>>>>>>
> >>>>>>>> As soon as I move to next commit (dated May 22nd, pushed and
> >>> deployed today) with id
> >>>>>>>>
> >>>>>>>> b51d003ed98aa7663465aad7c9a1b554b006cc7d
> >>>>>>>>
> >>>>>>>> I see the same behavior as with HEAD, e.g.
> >>>>>>>>
> >>>>>>>> 2b9b024f273d63e479a02cad751c28b8ef974ace
> >>>>>>>>
> >>>>>>>> I will try to take a closer look at the code.
> >>>>>>>> Regards.
> >>>>>>>>
> >>>>>>>> On 03/07/23 14:44, Mark Struberg wrote:
> >>>>>>>>> Hi Francesco!
> >>>>>>>>>
> >>>>>>>>> That's very valuable information!
> >>>>>>>>> Could you please do a git-bisect to find out which change did
> break
> >>> it?
> >>>>>>>>>
> >>>>>>>>> Actually the algorithm to select which classes are to be enhanced
> >>> should not have changed. Thus I'm really curious what I did break.
> >>>>>>>>>
> >>>>>>>>> Most times the amount of enhanced classes are the same but
> slightly
> >>> differ in the generated code. That might be due to now we really
> generate
> >>> Java11++ code whereas before we only generated Java 1.1 code (yes, not
> even
> >>> 1.5!)
> >>>>>>>>>
> >>>>>>>>> What I usually do is to have 2 checkouts. You might want to
> change
> >>> the pom GAV version of the working checkout to something something
> >>> different so you might have both in your ~/.m2/repository in parallel.
> >>>>>>>>>
> >>>>>>>>> The first thing I do is to compare the class files in
> >>> target/classes of both versions with 'compare with clipboard'.
> >>>>>>>>> I just open the class file in target/classes and let
> >>> Intellij/Netbeans do the decompilation for me.
> >>>>>>>>>
> >>>>>>>>> You might find some differences here in which case this needs
> >>> further investigation. Note that the order of the methods might have
> >>> changed.
> >>>>>>>>> Sometimes you will also see some 'cannot decompile' in the code
> >>> block. This happens when something in the bytecode is messed up. You
> can
> >>> also watch out for ClassNotFound or VerifyError, etc.
> >>>>>>>>>
> >>>>>>>>> If this is the case I then use
> >>>>>>>>> $> javap -c -p -constants
> >>> target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile
> >>>>>>>>> on both versions and check whether something is different.
> >>>>>>>>> Often it's a wrong offset in the stack calculation (e.g. ALOAD 1
> vs
> >>> ALOAD 2)
> >>>>>>>>>
> >>>>>>>>> Note that the 'old' Java 1.1 bytecode doesn't understand LDC of
> >>> Class constants (LoaD Constant). This was only introduced in Java 1.4
> or
> >>> 1.5 bytecode afair. Before that a dynamic Class.forName was called and
> >>> stored in a static variable. Ugly stuff, and happy to finally get rid
> of
> >>> all those hacks...
> >>>>>>>>>
> >>>>>>>>> Hope that helps to get you started. Feel free to ping me again
> over
> >>> here or at the #apachee ASF slack channel.
> >>>>>>>>>
> >>>>>>>>> txs and LieGrue,
> >>>>>>>>> strub
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò <
> >>> ilgrosso@apache.org>:
> >>>>>>>>>>
> >>>>>>>>>> From a very preliminary analysis it seems that just a subset of
> >>> the persistence-capable classes are effectively found and enhanced: now
> >>> [2], it used to be [3].
> >>>>>>>>>>
> >>>>>>>>>> I'll continue investigating.
> >>>>>>>>>>
> >>>>>>>>>> Regards.
> >>>>>>>>>>
> >>>>>>>>>> On 03/07/23 13:56, Francesco Chicchiriccò wrote:
> >>>>>>>>>>> Hi Mark,
> >>>>>>>>>>> the build on Syncope master branch, using OpenJPA
> 4.0.0-SNAPSHOT
> >>> is now failing - it used to work at least until last June 30th.
> >>>>>>>>>>>
> >>>>>>>>>>> The issues seem to be related to entity enhancement, which we
> >>> perform via openjpa-maven-plugin - see [1].
> >>>>>>>>>>>
> >>>>>>>>>>> Is there anything obvious we should be changing?
> >>>>>>>>>>> Meanwhile I'll try to dig to see if I am able to understand
> >>> something more.
> >>>>>>>>>>>
> >>>>>>>>>>> Regards.
> >>>>>>>>>>>
> >>>>>>>>>>> [1]
> >>>
> https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174
> >>>>>>>>>> [2] https://paste.apache.org/gxcm6
> >>>>>>>>>> [3] https://paste.apache.org/1g9k5
> >>>>>>>>>>
> >>>>>>>>>>> On 03/07/23 12:15, Mark Struberg wrote:
> >>>>>>>>>>>> Update: I've now merged the wip to our master and pushed it to
> >>> the ASF gitox repo.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think we already came pretty far and it looks really
> >>> promising. So there is a high chance that we succeed, although there is
> >>> still plenty of work in front of us.
> >>>>>>>>>>>>
> >>>>>>>>>>>> How it works:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Basically there are 2 sets of information in the PCEnhancer
> >>> right now:
> >>>>>>>>>>>>
> >>>>>>>>>>>> _managed: the BCClass version of the original class
> >>>>>>>>>>>> _pc: the BCClass of the generated/modified class. Might be the
> >>> same as _managed but in case of subclassing or interface case it is
> >>> different. In those cases _managed represents the original class and
> _pc
> >>> the subclass/concrete implementation.
> >>>>>>>>>>>>
> >>>>>>>>>>>> managed and pc are the ASM variants of _managed and _pc.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The main entry point for debugging into it is PCEnhancer#run()
> >>>>>>>>>>>>
> >>>>>>>>>>>> There are a few methods in ASMHelper to update from BCClass ->
> >>> ASM and vice versa:
> >>>>>>>>>>>>
> >>>>>>>>>>>> AsmHelper.readIntoBCClass(pc, _pc);
> >>>>>>>>>>>> takes all the information from the ASM pc, creates a class
> >>> byte[] and reads that back into the existing _pc instance.
> >>>>>>>>>>>>
> >>>>>>>>>>>> pc = AsmHelper.toClassNode(_pc);
> >>>>>>>>>>>> is the corresponding method in the other direction. from
> BCClass
> >>> to ASM.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Right now we have the following methods left to migrate in
> >>> PCEnhancer
> >>>>>>>>>>>> addAttachDetachCode();
> >>>>>>>>>>>> addSerializationCode();
> >>>>>>>>>>>> addCloningCode();
> >>>>>>>>>>>> runAuxiliaryEnhancers();
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'll be off for a week on holidays in Venice.
> >>>>>>>>>>>> If someone want's to join in with the effort then please go
> on!
> >>> You might want to compare the current PCEnhancer code with an older
> version
> >>> to get an idea how it can be done and to see some patterns I've used.
> >>>>>>>>>>>>
> >>>>>>>>>>>> As a quick hint: Java is basically a stack based virtual CPU.
> >>> The 'this' pointer is on position 0 of the stack, so ALOAD 0. After
> that
> >>> comes the parameters. The size of the param on the stack depends on the
> >>> type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st
> method
> >>> parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
> >>>>>>>>>>>> In case of a static method there is ofc no 'this' on the
> stack,
> >>> so the parameters start with zero offset.
> >>>>>>>>>>>> After all the parameters there are the unnamed 'local'
> >>> variables. I tried to consistently name them xxxVarPos.
> >>>>>>>>>>>> See also the various helper methods in ASMHelper. For example
> >>> for the various LOAD, RETURN, STORE operations which depends based on
> the
> >>> type (ALOAD vs ILOAD vs IALOAD etc)
> >>>>>>>>>>>> Also read the guide to ASM PDF from objectweb.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'll be mostly offline till Sunday, but will try to answer
> mails
> >>> if there are some questions.
> >>>>>>>>>>>>
> >>>>>>>>>>>> txs and LieGrue,
> >>>>>>>>>>>> strub
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <
> >>> mattrpav@gmail.com>:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> +1 thanks for tackling this, Mark!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ASM definitely more widely used going forward.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Matt
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg
> >>> <st...@yahoo.de.INVALID> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Small update:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I've worked on it over the last few weeks, and I'm getting
> >>> closer
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> contains the latest work on the PCEnhancer. Right in the
> >>> middle of replacing Serp with native ASM code.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Note that I also had to modify a few interfaces with a few
> >>> more to follow from BCClass to ASM ClassNode.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Happy to get some feedback!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If anybody wants to help with this effort I'm happy to also
> >>> push this feature branch to our ASF repo. It looks reasonably promising
> >>> already.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> LieGrue,
> >>>>>>>>>>>>>> strub
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg
> >>> <st...@yahoo.de.INVALID>:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Right now I'm trying to get rid of serp step by step.
> >>>>>>>>>>>>>>> The code is right now at my own github repo in the
> >>> fb_asmEnhance branch:
> >>>>>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The approach I took for now is to create a few methods in
> >>> AsmHelper to be able to move from BCClass -> ASM ClassWriter and the
> other
> >>> way around. That way we should be able to replace functionality part by
> >>> part but still keep all things afloat.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> For now I've started with the PCSubclassValidator.
> >>>>>>>>>>>>>>> Right now this evaluates the attributs using Serp plus ASM
> >>> and then compare the results.
> >>>>>>>>>>>>>>> If something is fishy, you'll see the following
> >>>>>>>>>>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField +
> "
> >>> " + field);
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It would be extremely helpful, if you could go through your
> >>> projects and let it run and report how it's going.
> >>>>>>>>>>>>>>> If you see that "MSX ASMTODO" somewhere then we know I
> messed
> >>> something up.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I'll gonna finally remove the BCClass handling from those
> >>> parts in a few days. Current commit is
> >>>>>>>>>>>>>>>
> >>>
> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> txs and LieGrue,
> >>>>>>>>>>>>>>> strub
> >>>>>
> >>>>
> >>>> --
> >>>> Francesco Chicchiriccò
> >>>>
> >>>> Tirasa - Open Source Excellence
> >>>> http://www.tirasa.net/
> >>>>
> >>>> Member at The Apache Software Foundation
> >>>> Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
> >>>> http://home.apache.org/~ilgrosso/
> >>>>
> >>>
> >>>
> >
>
>
Re: enhance without Serp - Update
Posted by Mark Struberg <st...@yahoo.de.INVALID>.
Francesco, I've now pushed this change. Can you please try it out? Looks good over here now.
Now I started the full Syncope build and hope all comes green again.
txs and LieGrue,
strub
> Am 03.07.2023 um 17:55 schrieb Mark Struberg <st...@yahoo.de>:
>
> Think I found the solution, but first want to run all tests in OpenJPA.
>
> I did use the standard ClassWriter in ASMHelper. This does use Class.forName.
> Changed ASMHelper to use the BCClassWriter you wrote some time ago. That should do the trick I hope.
>
> Gimme 10 minutes plz.
>
> LieGrue,
> strub
>
>
>> Am 03.07.2023 um 17:34 schrieb Romain Manni-Bucau <rm...@gmail.com>:
>>
>> @Francesco: can you *javap -v -cp $yourclasspath the.class.which.Fail* when
>> it works vs when it fails please?
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>>
>> Le lun. 3 juil. 2023 à 17:23, Mark Struberg <st...@yahoo.de.invalid> a
>> écrit :
>>
>>> Does not yet ring any bells, sorry :(
>>>
>>> Right now trying to fire up the enhancer manually in the debugger.
>>> Difference really is that we now do the ASM part way earlier, but still
>>> class per class.
>>> Before that it was done as part of the ASMAdapter.
>>>
>>> Still digging.
>>>
>>> LieGrue,
>>> strub
>>>
>>>
>>>> Am 03.07.2023 um 17:00 schrieb Francesco Chicchiriccò <
>>> ilgrosso@apache.org>:
>>>>
>>>> Adding some info.
>>>>
>>>> First of all, the class to which the error message is referring is not
>>> always the same.
>>>>
>>>> Secondly, at least about AccountPolicy as reported below, the original
>>> error message is stating
>>>>
>>>> Execution enhancer of goal
>>> org.apache.openjpa:openjpa-maven-plugin:4.0.0-SNAPSHOT:enhance failed: An
>>> error occurred while enhancing
>>> org.apache.syncope.core.persistence.jpa.entity.JPARealm. Exception message:
>>> Type org/apache/syncope/core/persistence/api/entity/policy/AccountPolicy
>>> not present:
>>> org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
>>>>
>>>> If you look at JPARealm:
>>>>
>>>>
>>> https://github.com/apache/syncope/blob/master/core/persistence-jpa/src/main/java/org/apache/syncope/core/persistence/jpa/entity/JPARealm.java#L82
>>>>
>>>> it declares
>>>>
>>>> @ManyToOne(fetch = FetchType.EAGER)
>>>> private JPAAccountPolicy accountPolicy;
>>>>
>>>> e.g. a reference to the JPA entity JPAAccountPolicy, not to the
>>> interface that this class implements, e.g. AccountPolicy
>>>>
>>>> It looks to me that somewhere the OpenJPA enhancement process is trying
>>> to resolve more than it should do.
>>>>
>>>> Does this make ringing any bell?
>>>>
>>>> Regards.
>>>>
>>>> On 03/07/23 16:32, Francesco Chicchiriccò wrote:
>>>>> On 03/07/23 16:27, Mark Struberg wrote:
>>>>>> Went forward and compiled the Syncope main branch.
>>>>>>
>>>>>> Getting this error when running with mvn -X
>>>>>>
>>>>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
>>>>>> at
>>> org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass
>>> (SelfFirstStrategy.java:50)
>>>>>> at
>>> org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass
>>> (ClassRealm.java:271)
>>>>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass
>>> (ClassRealm.java:247)
>>>>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass
>>> (ClassRealm.java:239)
>>>>>> at java.lang.Class.forName0 (Native Method)
>>>>>> at java.lang.Class.forName (Class.java:467)
>>>>>> at org.apache.xbean.asm9.ClassWriter.getCommonSuperClass
>>> (ClassWriter.java:1043)
>>>>>> at org.apache.xbean.asm9.SymbolTable.addMergedType
>>> (SymbolTable.java:1202)
>>>>>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1299)
>>>>>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1244)
>>>>>> at org.apache.xbean.asm9.MethodWriter.computeAllFrames
>>> (MethodWriter.java:1611)
>>>>>> at org.apache.xbean.asm9.MethodWriter.visitMaxs
>>> (MethodWriter.java:1547)
>>>>>> at org.apache.xbean.asm9.tree.MethodNode.accept
>>> (MethodNode.java:767)
>>>>>> at org.apache.xbean.asm9.tree.MethodNode.accept
>>> (MethodNode.java:647)
>>>>>> at org.apache.xbean.asm9.tree.ClassNode.accept
>>> (ClassNode.java:468)
>>>>>> at org.apache.openjpa.util.asm.AsmHelper.readIntoBCClass
>>> (AsmHelper.java:146)
>>>>>> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:606)
>>>>>> at org.apache.openjpa.enhance.PCEnhancer.run
>>> (PCEnhancer.java:5889)
>>>>>> at org.apache.openjpa.enhance.PCEnhancer.run
>>> (PCEnhancer.java:5829)
>>>>>> at org.apache.openjpa.enhance.PCEnhancer$1.run
>>> (PCEnhancer.java:5799)
>>>>>> at org.apache.openjpa.lib.conf.Configurations.launchRunnable
>>> (Configurations.java:760)
>>>>>> at
>>> org.apache.openjpa.lib.conf.Configurations.runAgainstAllAnchors
>>> (Configurations.java:745)
>>>>>>
>>>>>>
>>>>>> Is this the one you experience too?
>>>>>
>>>>> Yes, exactly.
>>>>>
>>>>> As you might have found out, AccountPolicy is an interface implemented
>>> by JPAAccountPolicy, which is the corresponding JPA entity.
>>>>> We take this approach for all JPA entities, actually.
>>>>>
>>>>> I thought the reason why AccountPolicy or JPAAccountPolicy are not
>>> found is that enhancement does not actually finds them.
>>>>>
>>>>>> Will take a look at it now.
>>>>>
>>>>> Thank you very much!
>>>>>
>>>>> Regards.
>>>>>>> Am 03.07.2023 um 15:49 schrieb Mark Struberg
>>> <st...@yahoo.de.INVALID>:
>>>>>>>
>>>>>>> Hi Francesco!
>>>>>>>
>>>>>>> Can you see which one of the classes fail to enhance?
>>>>>>>
>>>>>>> In which case you might try to run the enhancement of that very class
>>> manually in your IDE.
>>>>>>>
>>>>>>> Main class: org.apache.openjpa.enhance.PCEnhancer
>>>>>>> VMOptions:: -ea
>>> -Dopenjpa.ConnectionURL=jdbc:derby:target/database/openjpa-derby-database;create=true
>>> -Dopenjpa.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver
>>>>>>> or something similar for your connection
>>>>>>>
>>>>>>> Then you also need the parameter pointing to the class file which you
>>> want to enhance, e.g.
>>>>>>>
>>> target/test-classes/org/apache/openjpa/persistence/proxy/entities/Annuity.class
>>>>>>>
>>>>>>> Please keep me updated!
>>>>>>>
>>>>>>>
>>>>>>> LieGrue,
>>>>>>> strub
>>>>>>>
>>>>>>> PS: does it add value if we'd write all this up? You will likely know
>>> most of that stuff, but others who dig into OpenJPA do not.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Am 03.07.2023 um 15:28 schrieb Francesco Chicchiriccò <
>>> ilgrosso@apache.org>:
>>>>>>>>
>>>>>>>> Hi Mark,
>>>>>>>> thanks for your reply.
>>>>>>>>
>>>>>>>> So I am proceeding as follows.
>>>>>>>>
>>>>>>>> Took the executions at
>>>>>>>>
>>>>>>>> https://ci-builds.apache.org/job/OpenJPA/job/OpenJPA-master-deploy/
>>>>>>>>
>>>>>>>> (the Jenkins job which is publishing SNAPSHOT artifacts from master
>>> to ASF Maven repo) and saw that the artifacts with which the Syncope build
>>> succeeded last Friday are the one from May 16th, so the corresponding
>>> commit id is
>>>>>>>>
>>>>>>>> b238417dabeac935846fe1ce2fef28aafaeef205
>>>>>>>>
>>>>>>>> I confirm that building locally from that commit id everything is
>>> working.
>>>>>>>>
>>>>>>>> As soon as I move to next commit (dated May 22nd, pushed and
>>> deployed today) with id
>>>>>>>>
>>>>>>>> b51d003ed98aa7663465aad7c9a1b554b006cc7d
>>>>>>>>
>>>>>>>> I see the same behavior as with HEAD, e.g.
>>>>>>>>
>>>>>>>> 2b9b024f273d63e479a02cad751c28b8ef974ace
>>>>>>>>
>>>>>>>> I will try to take a closer look at the code.
>>>>>>>> Regards.
>>>>>>>>
>>>>>>>> On 03/07/23 14:44, Mark Struberg wrote:
>>>>>>>>> Hi Francesco!
>>>>>>>>>
>>>>>>>>> That's very valuable information!
>>>>>>>>> Could you please do a git-bisect to find out which change did break
>>> it?
>>>>>>>>>
>>>>>>>>> Actually the algorithm to select which classes are to be enhanced
>>> should not have changed. Thus I'm really curious what I did break.
>>>>>>>>>
>>>>>>>>> Most times the amount of enhanced classes are the same but slightly
>>> differ in the generated code. That might be due to now we really generate
>>> Java11++ code whereas before we only generated Java 1.1 code (yes, not even
>>> 1.5!)
>>>>>>>>>
>>>>>>>>> What I usually do is to have 2 checkouts. You might want to change
>>> the pom GAV version of the working checkout to something something
>>> different so you might have both in your ~/.m2/repository in parallel.
>>>>>>>>>
>>>>>>>>> The first thing I do is to compare the class files in
>>> target/classes of both versions with 'compare with clipboard'.
>>>>>>>>> I just open the class file in target/classes and let
>>> Intellij/Netbeans do the decompilation for me.
>>>>>>>>>
>>>>>>>>> You might find some differences here in which case this needs
>>> further investigation. Note that the order of the methods might have
>>> changed.
>>>>>>>>> Sometimes you will also see some 'cannot decompile' in the code
>>> block. This happens when something in the bytecode is messed up. You can
>>> also watch out for ClassNotFound or VerifyError, etc.
>>>>>>>>>
>>>>>>>>> If this is the case I then use
>>>>>>>>> $> javap -c -p -constants
>>> target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile
>>>>>>>>> on both versions and check whether something is different.
>>>>>>>>> Often it's a wrong offset in the stack calculation (e.g. ALOAD 1 vs
>>> ALOAD 2)
>>>>>>>>>
>>>>>>>>> Note that the 'old' Java 1.1 bytecode doesn't understand LDC of
>>> Class constants (LoaD Constant). This was only introduced in Java 1.4 or
>>> 1.5 bytecode afair. Before that a dynamic Class.forName was called and
>>> stored in a static variable. Ugly stuff, and happy to finally get rid of
>>> all those hacks...
>>>>>>>>>
>>>>>>>>> Hope that helps to get you started. Feel free to ping me again over
>>> here or at the #apachee ASF slack channel.
>>>>>>>>>
>>>>>>>>> txs and LieGrue,
>>>>>>>>> strub
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò <
>>> ilgrosso@apache.org>:
>>>>>>>>>>
>>>>>>>>>> From a very preliminary analysis it seems that just a subset of
>>> the persistence-capable classes are effectively found and enhanced: now
>>> [2], it used to be [3].
>>>>>>>>>>
>>>>>>>>>> I'll continue investigating.
>>>>>>>>>>
>>>>>>>>>> Regards.
>>>>>>>>>>
>>>>>>>>>> On 03/07/23 13:56, Francesco Chicchiriccò wrote:
>>>>>>>>>>> Hi Mark,
>>>>>>>>>>> the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT
>>> is now failing - it used to work at least until last June 30th.
>>>>>>>>>>>
>>>>>>>>>>> The issues seem to be related to entity enhancement, which we
>>> perform via openjpa-maven-plugin - see [1].
>>>>>>>>>>>
>>>>>>>>>>> Is there anything obvious we should be changing?
>>>>>>>>>>> Meanwhile I'll try to dig to see if I am able to understand
>>> something more.
>>>>>>>>>>>
>>>>>>>>>>> Regards.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>> https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174
>>>>>>>>>> [2] https://paste.apache.org/gxcm6
>>>>>>>>>> [3] https://paste.apache.org/1g9k5
>>>>>>>>>>
>>>>>>>>>>> On 03/07/23 12:15, Mark Struberg wrote:
>>>>>>>>>>>> Update: I've now merged the wip to our master and pushed it to
>>> the ASF gitox repo.
>>>>>>>>>>>>
>>>>>>>>>>>> I think we already came pretty far and it looks really
>>> promising. So there is a high chance that we succeed, although there is
>>> still plenty of work in front of us.
>>>>>>>>>>>>
>>>>>>>>>>>> How it works:
>>>>>>>>>>>>
>>>>>>>>>>>> Basically there are 2 sets of information in the PCEnhancer
>>> right now:
>>>>>>>>>>>>
>>>>>>>>>>>> _managed: the BCClass version of the original class
>>>>>>>>>>>> _pc: the BCClass of the generated/modified class. Might be the
>>> same as _managed but in case of subclassing or interface case it is
>>> different. In those cases _managed represents the original class and _pc
>>> the subclass/concrete implementation.
>>>>>>>>>>>>
>>>>>>>>>>>> managed and pc are the ASM variants of _managed and _pc.
>>>>>>>>>>>>
>>>>>>>>>>>> The main entry point for debugging into it is PCEnhancer#run()
>>>>>>>>>>>>
>>>>>>>>>>>> There are a few methods in ASMHelper to update from BCClass ->
>>> ASM and vice versa:
>>>>>>>>>>>>
>>>>>>>>>>>> AsmHelper.readIntoBCClass(pc, _pc);
>>>>>>>>>>>> takes all the information from the ASM pc, creates a class
>>> byte[] and reads that back into the existing _pc instance.
>>>>>>>>>>>>
>>>>>>>>>>>> pc = AsmHelper.toClassNode(_pc);
>>>>>>>>>>>> is the corresponding method in the other direction. from BCClass
>>> to ASM.
>>>>>>>>>>>>
>>>>>>>>>>>> Right now we have the following methods left to migrate in
>>> PCEnhancer
>>>>>>>>>>>> addAttachDetachCode();
>>>>>>>>>>>> addSerializationCode();
>>>>>>>>>>>> addCloningCode();
>>>>>>>>>>>> runAuxiliaryEnhancers();
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I'll be off for a week on holidays in Venice.
>>>>>>>>>>>> If someone want's to join in with the effort then please go on!
>>> You might want to compare the current PCEnhancer code with an older version
>>> to get an idea how it can be done and to see some patterns I've used.
>>>>>>>>>>>>
>>>>>>>>>>>> As a quick hint: Java is basically a stack based virtual CPU.
>>> The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that
>>> comes the parameters. The size of the param on the stack depends on the
>>> type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method
>>> parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
>>>>>>>>>>>> In case of a static method there is ofc no 'this' on the stack,
>>> so the parameters start with zero offset.
>>>>>>>>>>>> After all the parameters there are the unnamed 'local'
>>> variables. I tried to consistently name them xxxVarPos.
>>>>>>>>>>>> See also the various helper methods in ASMHelper. For example
>>> for the various LOAD, RETURN, STORE operations which depends based on the
>>> type (ALOAD vs ILOAD vs IALOAD etc)
>>>>>>>>>>>> Also read the guide to ASM PDF from objectweb.
>>>>>>>>>>>>
>>>>>>>>>>>> I'll be mostly offline till Sunday, but will try to answer mails
>>> if there are some questions.
>>>>>>>>>>>>
>>>>>>>>>>>> txs and LieGrue,
>>>>>>>>>>>> strub
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <
>>> mattrpav@gmail.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>> +1 thanks for tackling this, Mark!
>>>>>>>>>>>>>
>>>>>>>>>>>>> ASM definitely more widely used going forward.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Matt
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg
>>> <st...@yahoo.de.INVALID> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Small update:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've worked on it over the last few weeks, and I'm getting
>>> closer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> contains the latest work on the PCEnhancer. Right in the
>>> middle of replacing Serp with native ASM code.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Note that I also had to modify a few interfaces with a few
>>> more to follow from BCClass to ASM ClassNode.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Happy to get some feedback!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If anybody wants to help with this effort I'm happy to also
>>> push this feature branch to our ASF repo. It looks reasonably promising
>>> already.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LieGrue,
>>>>>>>>>>>>>> strub
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg
>>> <st...@yahoo.de.INVALID>:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Right now I'm trying to get rid of serp step by step.
>>>>>>>>>>>>>>> The code is right now at my own github repo in the
>>> fb_asmEnhance branch:
>>>>>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The approach I took for now is to create a few methods in
>>> AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other
>>> way around. That way we should be able to replace functionality part by
>>> part but still keep all things afloat.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For now I've started with the PCSubclassValidator.
>>>>>>>>>>>>>>> Right now this evaluates the attributs using Serp plus ASM
>>> and then compare the results.
>>>>>>>>>>>>>>> If something is fishy, you'll see the following
>>>>>>>>>>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + "
>>> " + field);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It would be extremely helpful, if you could go through your
>>> projects and let it run and report how it's going.
>>>>>>>>>>>>>>> If you see that "MSX ASMTODO" somewhere then we know I messed
>>> something up.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'll gonna finally remove the BCClass handling from those
>>> parts in a few days. Current commit is
>>>>>>>>>>>>>>>
>>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> txs and LieGrue,
>>>>>>>>>>>>>>> strub
>>>>>
>>>>
>>>> --
>>>> Francesco Chicchiriccò
>>>>
>>>> Tirasa - Open Source Excellence
>>>> http://www.tirasa.net/
>>>>
>>>> Member at The Apache Software Foundation
>>>> Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
>>>> http://home.apache.org/~ilgrosso/
>>>>
>>>
>>>
>
Re: enhance without Serp - Update
Posted by Mark Struberg <st...@yahoo.de.INVALID>.
Think I found the solution, but first want to run all tests in OpenJPA.
I did use the standard ClassWriter in ASMHelper. This does use Class.forName.
Changed ASMHelper to use the BCClassWriter you wrote some time ago. That should do the trick I hope.
Gimme 10 minutes plz.
LieGrue,
strub
> Am 03.07.2023 um 17:34 schrieb Romain Manni-Bucau <rm...@gmail.com>:
>
> @Francesco: can you *javap -v -cp $yourclasspath the.class.which.Fail* when
> it works vs when it fails please?
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
>
> Le lun. 3 juil. 2023 à 17:23, Mark Struberg <st...@yahoo.de.invalid> a
> écrit :
>
>> Does not yet ring any bells, sorry :(
>>
>> Right now trying to fire up the enhancer manually in the debugger.
>> Difference really is that we now do the ASM part way earlier, but still
>> class per class.
>> Before that it was done as part of the ASMAdapter.
>>
>> Still digging.
>>
>> LieGrue,
>> strub
>>
>>
>>> Am 03.07.2023 um 17:00 schrieb Francesco Chicchiriccò <
>> ilgrosso@apache.org>:
>>>
>>> Adding some info.
>>>
>>> First of all, the class to which the error message is referring is not
>> always the same.
>>>
>>> Secondly, at least about AccountPolicy as reported below, the original
>> error message is stating
>>>
>>> Execution enhancer of goal
>> org.apache.openjpa:openjpa-maven-plugin:4.0.0-SNAPSHOT:enhance failed: An
>> error occurred while enhancing
>> org.apache.syncope.core.persistence.jpa.entity.JPARealm. Exception message:
>> Type org/apache/syncope/core/persistence/api/entity/policy/AccountPolicy
>> not present:
>> org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
>>>
>>> If you look at JPARealm:
>>>
>>>
>> https://github.com/apache/syncope/blob/master/core/persistence-jpa/src/main/java/org/apache/syncope/core/persistence/jpa/entity/JPARealm.java#L82
>>>
>>> it declares
>>>
>>> @ManyToOne(fetch = FetchType.EAGER)
>>> private JPAAccountPolicy accountPolicy;
>>>
>>> e.g. a reference to the JPA entity JPAAccountPolicy, not to the
>> interface that this class implements, e.g. AccountPolicy
>>>
>>> It looks to me that somewhere the OpenJPA enhancement process is trying
>> to resolve more than it should do.
>>>
>>> Does this make ringing any bell?
>>>
>>> Regards.
>>>
>>> On 03/07/23 16:32, Francesco Chicchiriccò wrote:
>>>> On 03/07/23 16:27, Mark Struberg wrote:
>>>>> Went forward and compiled the Syncope main branch.
>>>>>
>>>>> Getting this error when running with mvn -X
>>>>>
>>>>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
>>>>> at
>> org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass
>> (SelfFirstStrategy.java:50)
>>>>> at
>> org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass
>> (ClassRealm.java:271)
>>>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass
>> (ClassRealm.java:247)
>>>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass
>> (ClassRealm.java:239)
>>>>> at java.lang.Class.forName0 (Native Method)
>>>>> at java.lang.Class.forName (Class.java:467)
>>>>> at org.apache.xbean.asm9.ClassWriter.getCommonSuperClass
>> (ClassWriter.java:1043)
>>>>> at org.apache.xbean.asm9.SymbolTable.addMergedType
>> (SymbolTable.java:1202)
>>>>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1299)
>>>>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1244)
>>>>> at org.apache.xbean.asm9.MethodWriter.computeAllFrames
>> (MethodWriter.java:1611)
>>>>> at org.apache.xbean.asm9.MethodWriter.visitMaxs
>> (MethodWriter.java:1547)
>>>>> at org.apache.xbean.asm9.tree.MethodNode.accept
>> (MethodNode.java:767)
>>>>> at org.apache.xbean.asm9.tree.MethodNode.accept
>> (MethodNode.java:647)
>>>>> at org.apache.xbean.asm9.tree.ClassNode.accept
>> (ClassNode.java:468)
>>>>> at org.apache.openjpa.util.asm.AsmHelper.readIntoBCClass
>> (AsmHelper.java:146)
>>>>> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:606)
>>>>> at org.apache.openjpa.enhance.PCEnhancer.run
>> (PCEnhancer.java:5889)
>>>>> at org.apache.openjpa.enhance.PCEnhancer.run
>> (PCEnhancer.java:5829)
>>>>> at org.apache.openjpa.enhance.PCEnhancer$1.run
>> (PCEnhancer.java:5799)
>>>>> at org.apache.openjpa.lib.conf.Configurations.launchRunnable
>> (Configurations.java:760)
>>>>> at
>> org.apache.openjpa.lib.conf.Configurations.runAgainstAllAnchors
>> (Configurations.java:745)
>>>>>
>>>>>
>>>>> Is this the one you experience too?
>>>>
>>>> Yes, exactly.
>>>>
>>>> As you might have found out, AccountPolicy is an interface implemented
>> by JPAAccountPolicy, which is the corresponding JPA entity.
>>>> We take this approach for all JPA entities, actually.
>>>>
>>>> I thought the reason why AccountPolicy or JPAAccountPolicy are not
>> found is that enhancement does not actually finds them.
>>>>
>>>>> Will take a look at it now.
>>>>
>>>> Thank you very much!
>>>>
>>>> Regards.
>>>>>> Am 03.07.2023 um 15:49 schrieb Mark Struberg
>> <st...@yahoo.de.INVALID>:
>>>>>>
>>>>>> Hi Francesco!
>>>>>>
>>>>>> Can you see which one of the classes fail to enhance?
>>>>>>
>>>>>> In which case you might try to run the enhancement of that very class
>> manually in your IDE.
>>>>>>
>>>>>> Main class: org.apache.openjpa.enhance.PCEnhancer
>>>>>> VMOptions:: -ea
>> -Dopenjpa.ConnectionURL=jdbc:derby:target/database/openjpa-derby-database;create=true
>> -Dopenjpa.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver
>>>>>> or something similar for your connection
>>>>>>
>>>>>> Then you also need the parameter pointing to the class file which you
>> want to enhance, e.g.
>>>>>>
>> target/test-classes/org/apache/openjpa/persistence/proxy/entities/Annuity.class
>>>>>>
>>>>>> Please keep me updated!
>>>>>>
>>>>>>
>>>>>> LieGrue,
>>>>>> strub
>>>>>>
>>>>>> PS: does it add value if we'd write all this up? You will likely know
>> most of that stuff, but others who dig into OpenJPA do not.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Am 03.07.2023 um 15:28 schrieb Francesco Chicchiriccò <
>> ilgrosso@apache.org>:
>>>>>>>
>>>>>>> Hi Mark,
>>>>>>> thanks for your reply.
>>>>>>>
>>>>>>> So I am proceeding as follows.
>>>>>>>
>>>>>>> Took the executions at
>>>>>>>
>>>>>>> https://ci-builds.apache.org/job/OpenJPA/job/OpenJPA-master-deploy/
>>>>>>>
>>>>>>> (the Jenkins job which is publishing SNAPSHOT artifacts from master
>> to ASF Maven repo) and saw that the artifacts with which the Syncope build
>> succeeded last Friday are the one from May 16th, so the corresponding
>> commit id is
>>>>>>>
>>>>>>> b238417dabeac935846fe1ce2fef28aafaeef205
>>>>>>>
>>>>>>> I confirm that building locally from that commit id everything is
>> working.
>>>>>>>
>>>>>>> As soon as I move to next commit (dated May 22nd, pushed and
>> deployed today) with id
>>>>>>>
>>>>>>> b51d003ed98aa7663465aad7c9a1b554b006cc7d
>>>>>>>
>>>>>>> I see the same behavior as with HEAD, e.g.
>>>>>>>
>>>>>>> 2b9b024f273d63e479a02cad751c28b8ef974ace
>>>>>>>
>>>>>>> I will try to take a closer look at the code.
>>>>>>> Regards.
>>>>>>>
>>>>>>> On 03/07/23 14:44, Mark Struberg wrote:
>>>>>>>> Hi Francesco!
>>>>>>>>
>>>>>>>> That's very valuable information!
>>>>>>>> Could you please do a git-bisect to find out which change did break
>> it?
>>>>>>>>
>>>>>>>> Actually the algorithm to select which classes are to be enhanced
>> should not have changed. Thus I'm really curious what I did break.
>>>>>>>>
>>>>>>>> Most times the amount of enhanced classes are the same but slightly
>> differ in the generated code. That might be due to now we really generate
>> Java11++ code whereas before we only generated Java 1.1 code (yes, not even
>> 1.5!)
>>>>>>>>
>>>>>>>> What I usually do is to have 2 checkouts. You might want to change
>> the pom GAV version of the working checkout to something something
>> different so you might have both in your ~/.m2/repository in parallel.
>>>>>>>>
>>>>>>>> The first thing I do is to compare the class files in
>> target/classes of both versions with 'compare with clipboard'.
>>>>>>>> I just open the class file in target/classes and let
>> Intellij/Netbeans do the decompilation for me.
>>>>>>>>
>>>>>>>> You might find some differences here in which case this needs
>> further investigation. Note that the order of the methods might have
>> changed.
>>>>>>>> Sometimes you will also see some 'cannot decompile' in the code
>> block. This happens when something in the bytecode is messed up. You can
>> also watch out for ClassNotFound or VerifyError, etc.
>>>>>>>>
>>>>>>>> If this is the case I then use
>>>>>>>> $> javap -c -p -constants
>> target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile
>>>>>>>> on both versions and check whether something is different.
>>>>>>>> Often it's a wrong offset in the stack calculation (e.g. ALOAD 1 vs
>> ALOAD 2)
>>>>>>>>
>>>>>>>> Note that the 'old' Java 1.1 bytecode doesn't understand LDC of
>> Class constants (LoaD Constant). This was only introduced in Java 1.4 or
>> 1.5 bytecode afair. Before that a dynamic Class.forName was called and
>> stored in a static variable. Ugly stuff, and happy to finally get rid of
>> all those hacks...
>>>>>>>>
>>>>>>>> Hope that helps to get you started. Feel free to ping me again over
>> here or at the #apachee ASF slack channel.
>>>>>>>>
>>>>>>>> txs and LieGrue,
>>>>>>>> strub
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò <
>> ilgrosso@apache.org>:
>>>>>>>>>
>>>>>>>>> From a very preliminary analysis it seems that just a subset of
>> the persistence-capable classes are effectively found and enhanced: now
>> [2], it used to be [3].
>>>>>>>>>
>>>>>>>>> I'll continue investigating.
>>>>>>>>>
>>>>>>>>> Regards.
>>>>>>>>>
>>>>>>>>> On 03/07/23 13:56, Francesco Chicchiriccò wrote:
>>>>>>>>>> Hi Mark,
>>>>>>>>>> the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT
>> is now failing - it used to work at least until last June 30th.
>>>>>>>>>>
>>>>>>>>>> The issues seem to be related to entity enhancement, which we
>> perform via openjpa-maven-plugin - see [1].
>>>>>>>>>>
>>>>>>>>>> Is there anything obvious we should be changing?
>>>>>>>>>> Meanwhile I'll try to dig to see if I am able to understand
>> something more.
>>>>>>>>>>
>>>>>>>>>> Regards.
>>>>>>>>>>
>>>>>>>>>> [1]
>> https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174
>>>>>>>>> [2] https://paste.apache.org/gxcm6
>>>>>>>>> [3] https://paste.apache.org/1g9k5
>>>>>>>>>
>>>>>>>>>> On 03/07/23 12:15, Mark Struberg wrote:
>>>>>>>>>>> Update: I've now merged the wip to our master and pushed it to
>> the ASF gitox repo.
>>>>>>>>>>>
>>>>>>>>>>> I think we already came pretty far and it looks really
>> promising. So there is a high chance that we succeed, although there is
>> still plenty of work in front of us.
>>>>>>>>>>>
>>>>>>>>>>> How it works:
>>>>>>>>>>>
>>>>>>>>>>> Basically there are 2 sets of information in the PCEnhancer
>> right now:
>>>>>>>>>>>
>>>>>>>>>>> _managed: the BCClass version of the original class
>>>>>>>>>>> _pc: the BCClass of the generated/modified class. Might be the
>> same as _managed but in case of subclassing or interface case it is
>> different. In those cases _managed represents the original class and _pc
>> the subclass/concrete implementation.
>>>>>>>>>>>
>>>>>>>>>>> managed and pc are the ASM variants of _managed and _pc.
>>>>>>>>>>>
>>>>>>>>>>> The main entry point for debugging into it is PCEnhancer#run()
>>>>>>>>>>>
>>>>>>>>>>> There are a few methods in ASMHelper to update from BCClass ->
>> ASM and vice versa:
>>>>>>>>>>>
>>>>>>>>>>> AsmHelper.readIntoBCClass(pc, _pc);
>>>>>>>>>>> takes all the information from the ASM pc, creates a class
>> byte[] and reads that back into the existing _pc instance.
>>>>>>>>>>>
>>>>>>>>>>> pc = AsmHelper.toClassNode(_pc);
>>>>>>>>>>> is the corresponding method in the other direction. from BCClass
>> to ASM.
>>>>>>>>>>>
>>>>>>>>>>> Right now we have the following methods left to migrate in
>> PCEnhancer
>>>>>>>>>>> addAttachDetachCode();
>>>>>>>>>>> addSerializationCode();
>>>>>>>>>>> addCloningCode();
>>>>>>>>>>> runAuxiliaryEnhancers();
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'll be off for a week on holidays in Venice.
>>>>>>>>>>> If someone want's to join in with the effort then please go on!
>> You might want to compare the current PCEnhancer code with an older version
>> to get an idea how it can be done and to see some patterns I've used.
>>>>>>>>>>>
>>>>>>>>>>> As a quick hint: Java is basically a stack based virtual CPU.
>> The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that
>> comes the parameters. The size of the param on the stack depends on the
>> type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method
>> parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
>>>>>>>>>>> In case of a static method there is ofc no 'this' on the stack,
>> so the parameters start with zero offset.
>>>>>>>>>>> After all the parameters there are the unnamed 'local'
>> variables. I tried to consistently name them xxxVarPos.
>>>>>>>>>>> See also the various helper methods in ASMHelper. For example
>> for the various LOAD, RETURN, STORE operations which depends based on the
>> type (ALOAD vs ILOAD vs IALOAD etc)
>>>>>>>>>>> Also read the guide to ASM PDF from objectweb.
>>>>>>>>>>>
>>>>>>>>>>> I'll be mostly offline till Sunday, but will try to answer mails
>> if there are some questions.
>>>>>>>>>>>
>>>>>>>>>>> txs and LieGrue,
>>>>>>>>>>> strub
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <
>> mattrpav@gmail.com>:
>>>>>>>>>>>>
>>>>>>>>>>>> +1 thanks for tackling this, Mark!
>>>>>>>>>>>>
>>>>>>>>>>>> ASM definitely more widely used going forward.
>>>>>>>>>>>>
>>>>>>>>>>>> -Matt
>>>>>>>>>>>>
>>>>>>>>>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg
>> <st...@yahoo.de.INVALID> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Small update:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've worked on it over the last few weeks, and I'm getting
>> closer
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>>>>>>
>>>>>>>>>>>>> contains the latest work on the PCEnhancer. Right in the
>> middle of replacing Serp with native ASM code.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Note that I also had to modify a few interfaces with a few
>> more to follow from BCClass to ASM ClassNode.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Happy to get some feedback!
>>>>>>>>>>>>>
>>>>>>>>>>>>> If anybody wants to help with this effort I'm happy to also
>> push this feature branch to our ASF repo. It looks reasonably promising
>> already.
>>>>>>>>>>>>>
>>>>>>>>>>>>> LieGrue,
>>>>>>>>>>>>> strub
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg
>> <st...@yahoo.de.INVALID>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Right now I'm trying to get rid of serp step by step.
>>>>>>>>>>>>>> The code is right now at my own github repo in the
>> fb_asmEnhance branch:
>>>>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The approach I took for now is to create a few methods in
>> AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other
>> way around. That way we should be able to replace functionality part by
>> part but still keep all things afloat.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For now I've started with the PCSubclassValidator.
>>>>>>>>>>>>>> Right now this evaluates the attributs using Serp plus ASM
>> and then compare the results.
>>>>>>>>>>>>>> If something is fishy, you'll see the following
>>>>>>>>>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + "
>> " + field);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It would be extremely helpful, if you could go through your
>> projects and let it run and report how it's going.
>>>>>>>>>>>>>> If you see that "MSX ASMTODO" somewhere then we know I messed
>> something up.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'll gonna finally remove the BCClass handling from those
>> parts in a few days. Current commit is
>>>>>>>>>>>>>>
>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> txs and LieGrue,
>>>>>>>>>>>>>> strub
>>>>
>>>
>>> --
>>> Francesco Chicchiriccò
>>>
>>> Tirasa - Open Source Excellence
>>> http://www.tirasa.net/
>>>
>>> Member at The Apache Software Foundation
>>> Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
>>> http://home.apache.org/~ilgrosso/
>>>
>>
>>
Re: enhance without Serp - Update
Posted by Romain Manni-Bucau <rm...@gmail.com>.
@Francesco: can you *javap -v -cp $yourclasspath the.class.which.Fail* when
it works vs when it fails please?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le lun. 3 juil. 2023 à 17:23, Mark Struberg <st...@yahoo.de.invalid> a
écrit :
> Does not yet ring any bells, sorry :(
>
> Right now trying to fire up the enhancer manually in the debugger.
> Difference really is that we now do the ASM part way earlier, but still
> class per class.
> Before that it was done as part of the ASMAdapter.
>
> Still digging.
>
> LieGrue,
> strub
>
>
> > Am 03.07.2023 um 17:00 schrieb Francesco Chicchiriccò <
> ilgrosso@apache.org>:
> >
> > Adding some info.
> >
> > First of all, the class to which the error message is referring is not
> always the same.
> >
> > Secondly, at least about AccountPolicy as reported below, the original
> error message is stating
> >
> > Execution enhancer of goal
> org.apache.openjpa:openjpa-maven-plugin:4.0.0-SNAPSHOT:enhance failed: An
> error occurred while enhancing
> org.apache.syncope.core.persistence.jpa.entity.JPARealm. Exception message:
> Type org/apache/syncope/core/persistence/api/entity/policy/AccountPolicy
> not present:
> org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
> >
> > If you look at JPARealm:
> >
> >
> https://github.com/apache/syncope/blob/master/core/persistence-jpa/src/main/java/org/apache/syncope/core/persistence/jpa/entity/JPARealm.java#L82
> >
> > it declares
> >
> > @ManyToOne(fetch = FetchType.EAGER)
> > private JPAAccountPolicy accountPolicy;
> >
> > e.g. a reference to the JPA entity JPAAccountPolicy, not to the
> interface that this class implements, e.g. AccountPolicy
> >
> > It looks to me that somewhere the OpenJPA enhancement process is trying
> to resolve more than it should do.
> >
> > Does this make ringing any bell?
> >
> > Regards.
> >
> > On 03/07/23 16:32, Francesco Chicchiriccò wrote:
> >> On 03/07/23 16:27, Mark Struberg wrote:
> >>> Went forward and compiled the Syncope main branch.
> >>>
> >>> Getting this error when running with mvn -X
> >>>
> >>> Caused by: java.lang.ClassNotFoundException:
> org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
> >>> at
> org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass
> (SelfFirstStrategy.java:50)
> >>> at
> org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass
> (ClassRealm.java:271)
> >>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass
> (ClassRealm.java:247)
> >>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass
> (ClassRealm.java:239)
> >>> at java.lang.Class.forName0 (Native Method)
> >>> at java.lang.Class.forName (Class.java:467)
> >>> at org.apache.xbean.asm9.ClassWriter.getCommonSuperClass
> (ClassWriter.java:1043)
> >>> at org.apache.xbean.asm9.SymbolTable.addMergedType
> (SymbolTable.java:1202)
> >>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1299)
> >>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1244)
> >>> at org.apache.xbean.asm9.MethodWriter.computeAllFrames
> (MethodWriter.java:1611)
> >>> at org.apache.xbean.asm9.MethodWriter.visitMaxs
> (MethodWriter.java:1547)
> >>> at org.apache.xbean.asm9.tree.MethodNode.accept
> (MethodNode.java:767)
> >>> at org.apache.xbean.asm9.tree.MethodNode.accept
> (MethodNode.java:647)
> >>> at org.apache.xbean.asm9.tree.ClassNode.accept
> (ClassNode.java:468)
> >>> at org.apache.openjpa.util.asm.AsmHelper.readIntoBCClass
> (AsmHelper.java:146)
> >>> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:606)
> >>> at org.apache.openjpa.enhance.PCEnhancer.run
> (PCEnhancer.java:5889)
> >>> at org.apache.openjpa.enhance.PCEnhancer.run
> (PCEnhancer.java:5829)
> >>> at org.apache.openjpa.enhance.PCEnhancer$1.run
> (PCEnhancer.java:5799)
> >>> at org.apache.openjpa.lib.conf.Configurations.launchRunnable
> (Configurations.java:760)
> >>> at
> org.apache.openjpa.lib.conf.Configurations.runAgainstAllAnchors
> (Configurations.java:745)
> >>>
> >>>
> >>> Is this the one you experience too?
> >>
> >> Yes, exactly.
> >>
> >> As you might have found out, AccountPolicy is an interface implemented
> by JPAAccountPolicy, which is the corresponding JPA entity.
> >> We take this approach for all JPA entities, actually.
> >>
> >> I thought the reason why AccountPolicy or JPAAccountPolicy are not
> found is that enhancement does not actually finds them.
> >>
> >>> Will take a look at it now.
> >>
> >> Thank you very much!
> >>
> >> Regards.
> >>>> Am 03.07.2023 um 15:49 schrieb Mark Struberg
> <st...@yahoo.de.INVALID>:
> >>>>
> >>>> Hi Francesco!
> >>>>
> >>>> Can you see which one of the classes fail to enhance?
> >>>>
> >>>> In which case you might try to run the enhancement of that very class
> manually in your IDE.
> >>>>
> >>>> Main class: org.apache.openjpa.enhance.PCEnhancer
> >>>> VMOptions:: -ea
> -Dopenjpa.ConnectionURL=jdbc:derby:target/database/openjpa-derby-database;create=true
> -Dopenjpa.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver
> >>>> or something similar for your connection
> >>>>
> >>>> Then you also need the parameter pointing to the class file which you
> want to enhance, e.g.
> >>>>
> target/test-classes/org/apache/openjpa/persistence/proxy/entities/Annuity.class
> >>>>
> >>>> Please keep me updated!
> >>>>
> >>>>
> >>>> LieGrue,
> >>>> strub
> >>>>
> >>>> PS: does it add value if we'd write all this up? You will likely know
> most of that stuff, but others who dig into OpenJPA do not.
> >>>>
> >>>>
> >>>>
> >>>>> Am 03.07.2023 um 15:28 schrieb Francesco Chicchiriccò <
> ilgrosso@apache.org>:
> >>>>>
> >>>>> Hi Mark,
> >>>>> thanks for your reply.
> >>>>>
> >>>>> So I am proceeding as follows.
> >>>>>
> >>>>> Took the executions at
> >>>>>
> >>>>> https://ci-builds.apache.org/job/OpenJPA/job/OpenJPA-master-deploy/
> >>>>>
> >>>>> (the Jenkins job which is publishing SNAPSHOT artifacts from master
> to ASF Maven repo) and saw that the artifacts with which the Syncope build
> succeeded last Friday are the one from May 16th, so the corresponding
> commit id is
> >>>>>
> >>>>> b238417dabeac935846fe1ce2fef28aafaeef205
> >>>>>
> >>>>> I confirm that building locally from that commit id everything is
> working.
> >>>>>
> >>>>> As soon as I move to next commit (dated May 22nd, pushed and
> deployed today) with id
> >>>>>
> >>>>> b51d003ed98aa7663465aad7c9a1b554b006cc7d
> >>>>>
> >>>>> I see the same behavior as with HEAD, e.g.
> >>>>>
> >>>>> 2b9b024f273d63e479a02cad751c28b8ef974ace
> >>>>>
> >>>>> I will try to take a closer look at the code.
> >>>>> Regards.
> >>>>>
> >>>>> On 03/07/23 14:44, Mark Struberg wrote:
> >>>>>> Hi Francesco!
> >>>>>>
> >>>>>> That's very valuable information!
> >>>>>> Could you please do a git-bisect to find out which change did break
> it?
> >>>>>>
> >>>>>> Actually the algorithm to select which classes are to be enhanced
> should not have changed. Thus I'm really curious what I did break.
> >>>>>>
> >>>>>> Most times the amount of enhanced classes are the same but slightly
> differ in the generated code. That might be due to now we really generate
> Java11++ code whereas before we only generated Java 1.1 code (yes, not even
> 1.5!)
> >>>>>>
> >>>>>> What I usually do is to have 2 checkouts. You might want to change
> the pom GAV version of the working checkout to something something
> different so you might have both in your ~/.m2/repository in parallel.
> >>>>>>
> >>>>>> The first thing I do is to compare the class files in
> target/classes of both versions with 'compare with clipboard'.
> >>>>>> I just open the class file in target/classes and let
> Intellij/Netbeans do the decompilation for me.
> >>>>>>
> >>>>>> You might find some differences here in which case this needs
> further investigation. Note that the order of the methods might have
> changed.
> >>>>>> Sometimes you will also see some 'cannot decompile' in the code
> block. This happens when something in the bytecode is messed up. You can
> also watch out for ClassNotFound or VerifyError, etc.
> >>>>>>
> >>>>>> If this is the case I then use
> >>>>>> $> javap -c -p -constants
> target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile
> >>>>>> on both versions and check whether something is different.
> >>>>>> Often it's a wrong offset in the stack calculation (e.g. ALOAD 1 vs
> ALOAD 2)
> >>>>>>
> >>>>>> Note that the 'old' Java 1.1 bytecode doesn't understand LDC of
> Class constants (LoaD Constant). This was only introduced in Java 1.4 or
> 1.5 bytecode afair. Before that a dynamic Class.forName was called and
> stored in a static variable. Ugly stuff, and happy to finally get rid of
> all those hacks...
> >>>>>>
> >>>>>> Hope that helps to get you started. Feel free to ping me again over
> here or at the #apachee ASF slack channel.
> >>>>>>
> >>>>>> txs and LieGrue,
> >>>>>> strub
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò <
> ilgrosso@apache.org>:
> >>>>>>>
> >>>>>>> From a very preliminary analysis it seems that just a subset of
> the persistence-capable classes are effectively found and enhanced: now
> [2], it used to be [3].
> >>>>>>>
> >>>>>>> I'll continue investigating.
> >>>>>>>
> >>>>>>> Regards.
> >>>>>>>
> >>>>>>> On 03/07/23 13:56, Francesco Chicchiriccò wrote:
> >>>>>>>> Hi Mark,
> >>>>>>>> the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT
> is now failing - it used to work at least until last June 30th.
> >>>>>>>>
> >>>>>>>> The issues seem to be related to entity enhancement, which we
> perform via openjpa-maven-plugin - see [1].
> >>>>>>>>
> >>>>>>>> Is there anything obvious we should be changing?
> >>>>>>>> Meanwhile I'll try to dig to see if I am able to understand
> something more.
> >>>>>>>>
> >>>>>>>> Regards.
> >>>>>>>>
> >>>>>>>> [1]
> https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174
> >>>>>>> [2] https://paste.apache.org/gxcm6
> >>>>>>> [3] https://paste.apache.org/1g9k5
> >>>>>>>
> >>>>>>>> On 03/07/23 12:15, Mark Struberg wrote:
> >>>>>>>>> Update: I've now merged the wip to our master and pushed it to
> the ASF gitox repo.
> >>>>>>>>>
> >>>>>>>>> I think we already came pretty far and it looks really
> promising. So there is a high chance that we succeed, although there is
> still plenty of work in front of us.
> >>>>>>>>>
> >>>>>>>>> How it works:
> >>>>>>>>>
> >>>>>>>>> Basically there are 2 sets of information in the PCEnhancer
> right now:
> >>>>>>>>>
> >>>>>>>>> _managed: the BCClass version of the original class
> >>>>>>>>> _pc: the BCClass of the generated/modified class. Might be the
> same as _managed but in case of subclassing or interface case it is
> different. In those cases _managed represents the original class and _pc
> the subclass/concrete implementation.
> >>>>>>>>>
> >>>>>>>>> managed and pc are the ASM variants of _managed and _pc.
> >>>>>>>>>
> >>>>>>>>> The main entry point for debugging into it is PCEnhancer#run()
> >>>>>>>>>
> >>>>>>>>> There are a few methods in ASMHelper to update from BCClass ->
> ASM and vice versa:
> >>>>>>>>>
> >>>>>>>>> AsmHelper.readIntoBCClass(pc, _pc);
> >>>>>>>>> takes all the information from the ASM pc, creates a class
> byte[] and reads that back into the existing _pc instance.
> >>>>>>>>>
> >>>>>>>>> pc = AsmHelper.toClassNode(_pc);
> >>>>>>>>> is the corresponding method in the other direction. from BCClass
> to ASM.
> >>>>>>>>>
> >>>>>>>>> Right now we have the following methods left to migrate in
> PCEnhancer
> >>>>>>>>> addAttachDetachCode();
> >>>>>>>>> addSerializationCode();
> >>>>>>>>> addCloningCode();
> >>>>>>>>> runAuxiliaryEnhancers();
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I'll be off for a week on holidays in Venice.
> >>>>>>>>> If someone want's to join in with the effort then please go on!
> You might want to compare the current PCEnhancer code with an older version
> to get an idea how it can be done and to see some patterns I've used.
> >>>>>>>>>
> >>>>>>>>> As a quick hint: Java is basically a stack based virtual CPU.
> The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that
> comes the parameters. The size of the param on the stack depends on the
> type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method
> parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
> >>>>>>>>> In case of a static method there is ofc no 'this' on the stack,
> so the parameters start with zero offset.
> >>>>>>>>> After all the parameters there are the unnamed 'local'
> variables. I tried to consistently name them xxxVarPos.
> >>>>>>>>> See also the various helper methods in ASMHelper. For example
> for the various LOAD, RETURN, STORE operations which depends based on the
> type (ALOAD vs ILOAD vs IALOAD etc)
> >>>>>>>>> Also read the guide to ASM PDF from objectweb.
> >>>>>>>>>
> >>>>>>>>> I'll be mostly offline till Sunday, but will try to answer mails
> if there are some questions.
> >>>>>>>>>
> >>>>>>>>> txs and LieGrue,
> >>>>>>>>> strub
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <
> mattrpav@gmail.com>:
> >>>>>>>>>>
> >>>>>>>>>> +1 thanks for tackling this, Mark!
> >>>>>>>>>>
> >>>>>>>>>> ASM definitely more widely used going forward.
> >>>>>>>>>>
> >>>>>>>>>> -Matt
> >>>>>>>>>>
> >>>>>>>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg
> <st...@yahoo.de.INVALID> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Small update:
> >>>>>>>>>>>
> >>>>>>>>>>> I've worked on it over the last few weeks, and I'm getting
> closer
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
> >>>>>>>>>>>
> >>>>>>>>>>> contains the latest work on the PCEnhancer. Right in the
> middle of replacing Serp with native ASM code.
> >>>>>>>>>>>
> >>>>>>>>>>> Note that I also had to modify a few interfaces with a few
> more to follow from BCClass to ASM ClassNode.
> >>>>>>>>>>>
> >>>>>>>>>>> Happy to get some feedback!
> >>>>>>>>>>>
> >>>>>>>>>>> If anybody wants to help with this effort I'm happy to also
> push this feature branch to our ASF repo. It looks reasonably promising
> already.
> >>>>>>>>>>>
> >>>>>>>>>>> LieGrue,
> >>>>>>>>>>> strub
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg
> <st...@yahoo.de.INVALID>:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi!
> >>>>>>>>>>>>
> >>>>>>>>>>>> Right now I'm trying to get rid of serp step by step.
> >>>>>>>>>>>> The code is right now at my own github repo in the
> fb_asmEnhance branch:
> >>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> The approach I took for now is to create a few methods in
> AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other
> way around. That way we should be able to replace functionality part by
> part but still keep all things afloat.
> >>>>>>>>>>>>
> >>>>>>>>>>>> For now I've started with the PCSubclassValidator.
> >>>>>>>>>>>> Right now this evaluates the attributs using Serp plus ASM
> and then compare the results.
> >>>>>>>>>>>> If something is fishy, you'll see the following
> >>>>>>>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + "
> " + field);
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> It would be extremely helpful, if you could go through your
> projects and let it run and report how it's going.
> >>>>>>>>>>>> If you see that "MSX ASMTODO" somewhere then we know I messed
> something up.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'll gonna finally remove the BCClass handling from those
> parts in a few days. Current commit is
> >>>>>>>>>>>>
> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
> >>>>>>>>>>>>
> >>>>>>>>>>>> txs and LieGrue,
> >>>>>>>>>>>> strub
> >>
> >
> > --
> > Francesco Chicchiriccò
> >
> > Tirasa - Open Source Excellence
> > http://www.tirasa.net/
> >
> > Member at The Apache Software Foundation
> > Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
> > http://home.apache.org/~ilgrosso/
> >
>
>
Re: enhance without Serp - Update
Posted by Mark Struberg <st...@yahoo.de.INVALID>.
Does not yet ring any bells, sorry :(
Right now trying to fire up the enhancer manually in the debugger.
Difference really is that we now do the ASM part way earlier, but still class per class.
Before that it was done as part of the ASMAdapter.
Still digging.
LieGrue,
strub
> Am 03.07.2023 um 17:00 schrieb Francesco Chicchiriccò <il...@apache.org>:
>
> Adding some info.
>
> First of all, the class to which the error message is referring is not always the same.
>
> Secondly, at least about AccountPolicy as reported below, the original error message is stating
>
> Execution enhancer of goal org.apache.openjpa:openjpa-maven-plugin:4.0.0-SNAPSHOT:enhance failed: An error occurred while enhancing org.apache.syncope.core.persistence.jpa.entity.JPARealm. Exception message: Type org/apache/syncope/core/persistence/api/entity/policy/AccountPolicy not present: org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
>
> If you look at JPARealm:
>
> https://github.com/apache/syncope/blob/master/core/persistence-jpa/src/main/java/org/apache/syncope/core/persistence/jpa/entity/JPARealm.java#L82
>
> it declares
>
> @ManyToOne(fetch = FetchType.EAGER)
> private JPAAccountPolicy accountPolicy;
>
> e.g. a reference to the JPA entity JPAAccountPolicy, not to the interface that this class implements, e.g. AccountPolicy
>
> It looks to me that somewhere the OpenJPA enhancement process is trying to resolve more than it should do.
>
> Does this make ringing any bell?
>
> Regards.
>
> On 03/07/23 16:32, Francesco Chicchiriccò wrote:
>> On 03/07/23 16:27, Mark Struberg wrote:
>>> Went forward and compiled the Syncope main branch.
>>>
>>> Getting this error when running with mvn -X
>>>
>>> Caused by: java.lang.ClassNotFoundException: org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
>>> at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass (SelfFirstStrategy.java:50)
>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass (ClassRealm.java:271)
>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:247)
>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:239)
>>> at java.lang.Class.forName0 (Native Method)
>>> at java.lang.Class.forName (Class.java:467)
>>> at org.apache.xbean.asm9.ClassWriter.getCommonSuperClass (ClassWriter.java:1043)
>>> at org.apache.xbean.asm9.SymbolTable.addMergedType (SymbolTable.java:1202)
>>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1299)
>>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1244)
>>> at org.apache.xbean.asm9.MethodWriter.computeAllFrames (MethodWriter.java:1611)
>>> at org.apache.xbean.asm9.MethodWriter.visitMaxs (MethodWriter.java:1547)
>>> at org.apache.xbean.asm9.tree.MethodNode.accept (MethodNode.java:767)
>>> at org.apache.xbean.asm9.tree.MethodNode.accept (MethodNode.java:647)
>>> at org.apache.xbean.asm9.tree.ClassNode.accept (ClassNode.java:468)
>>> at org.apache.openjpa.util.asm.AsmHelper.readIntoBCClass (AsmHelper.java:146)
>>> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:606)
>>> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:5889)
>>> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:5829)
>>> at org.apache.openjpa.enhance.PCEnhancer$1.run (PCEnhancer.java:5799)
>>> at org.apache.openjpa.lib.conf.Configurations.launchRunnable (Configurations.java:760)
>>> at org.apache.openjpa.lib.conf.Configurations.runAgainstAllAnchors (Configurations.java:745)
>>>
>>>
>>> Is this the one you experience too?
>>
>> Yes, exactly.
>>
>> As you might have found out, AccountPolicy is an interface implemented by JPAAccountPolicy, which is the corresponding JPA entity.
>> We take this approach for all JPA entities, actually.
>>
>> I thought the reason why AccountPolicy or JPAAccountPolicy are not found is that enhancement does not actually finds them.
>>
>>> Will take a look at it now.
>>
>> Thank you very much!
>>
>> Regards.
>>>> Am 03.07.2023 um 15:49 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>>>
>>>> Hi Francesco!
>>>>
>>>> Can you see which one of the classes fail to enhance?
>>>>
>>>> In which case you might try to run the enhancement of that very class manually in your IDE.
>>>>
>>>> Main class: org.apache.openjpa.enhance.PCEnhancer
>>>> VMOptions:: -ea -Dopenjpa.ConnectionURL=jdbc:derby:target/database/openjpa-derby-database;create=true -Dopenjpa.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver
>>>> or something similar for your connection
>>>>
>>>> Then you also need the parameter pointing to the class file which you want to enhance, e.g.
>>>> target/test-classes/org/apache/openjpa/persistence/proxy/entities/Annuity.class
>>>>
>>>> Please keep me updated!
>>>>
>>>>
>>>> LieGrue,
>>>> strub
>>>>
>>>> PS: does it add value if we'd write all this up? You will likely know most of that stuff, but others who dig into OpenJPA do not.
>>>>
>>>>
>>>>
>>>>> Am 03.07.2023 um 15:28 schrieb Francesco Chicchiriccò <il...@apache.org>:
>>>>>
>>>>> Hi Mark,
>>>>> thanks for your reply.
>>>>>
>>>>> So I am proceeding as follows.
>>>>>
>>>>> Took the executions at
>>>>>
>>>>> https://ci-builds.apache.org/job/OpenJPA/job/OpenJPA-master-deploy/
>>>>>
>>>>> (the Jenkins job which is publishing SNAPSHOT artifacts from master to ASF Maven repo) and saw that the artifacts with which the Syncope build succeeded last Friday are the one from May 16th, so the corresponding commit id is
>>>>>
>>>>> b238417dabeac935846fe1ce2fef28aafaeef205
>>>>>
>>>>> I confirm that building locally from that commit id everything is working.
>>>>>
>>>>> As soon as I move to next commit (dated May 22nd, pushed and deployed today) with id
>>>>>
>>>>> b51d003ed98aa7663465aad7c9a1b554b006cc7d
>>>>>
>>>>> I see the same behavior as with HEAD, e.g.
>>>>>
>>>>> 2b9b024f273d63e479a02cad751c28b8ef974ace
>>>>>
>>>>> I will try to take a closer look at the code.
>>>>> Regards.
>>>>>
>>>>> On 03/07/23 14:44, Mark Struberg wrote:
>>>>>> Hi Francesco!
>>>>>>
>>>>>> That's very valuable information!
>>>>>> Could you please do a git-bisect to find out which change did break it?
>>>>>>
>>>>>> Actually the algorithm to select which classes are to be enhanced should not have changed. Thus I'm really curious what I did break.
>>>>>>
>>>>>> Most times the amount of enhanced classes are the same but slightly differ in the generated code. That might be due to now we really generate Java11++ code whereas before we only generated Java 1.1 code (yes, not even 1.5!)
>>>>>>
>>>>>> What I usually do is to have 2 checkouts. You might want to change the pom GAV version of the working checkout to something something different so you might have both in your ~/.m2/repository in parallel.
>>>>>>
>>>>>> The first thing I do is to compare the class files in target/classes of both versions with 'compare with clipboard'.
>>>>>> I just open the class file in target/classes and let Intellij/Netbeans do the decompilation for me.
>>>>>>
>>>>>> You might find some differences here in which case this needs further investigation. Note that the order of the methods might have changed.
>>>>>> Sometimes you will also see some 'cannot decompile' in the code block. This happens when something in the bytecode is messed up. You can also watch out for ClassNotFound or VerifyError, etc.
>>>>>>
>>>>>> If this is the case I then use
>>>>>> $> javap -c -p -constants target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile
>>>>>> on both versions and check whether something is different.
>>>>>> Often it's a wrong offset in the stack calculation (e.g. ALOAD 1 vs ALOAD 2)
>>>>>>
>>>>>> Note that the 'old' Java 1.1 bytecode doesn't understand LDC of Class constants (LoaD Constant). This was only introduced in Java 1.4 or 1.5 bytecode afair. Before that a dynamic Class.forName was called and stored in a static variable. Ugly stuff, and happy to finally get rid of all those hacks...
>>>>>>
>>>>>> Hope that helps to get you started. Feel free to ping me again over here or at the #apachee ASF slack channel.
>>>>>>
>>>>>> txs and LieGrue,
>>>>>> strub
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò <il...@apache.org>:
>>>>>>>
>>>>>>> From a very preliminary analysis it seems that just a subset of the persistence-capable classes are effectively found and enhanced: now [2], it used to be [3].
>>>>>>>
>>>>>>> I'll continue investigating.
>>>>>>>
>>>>>>> Regards.
>>>>>>>
>>>>>>> On 03/07/23 13:56, Francesco Chicchiriccò wrote:
>>>>>>>> Hi Mark,
>>>>>>>> the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT is now failing - it used to work at least until last June 30th.
>>>>>>>>
>>>>>>>> The issues seem to be related to entity enhancement, which we perform via openjpa-maven-plugin - see [1].
>>>>>>>>
>>>>>>>> Is there anything obvious we should be changing?
>>>>>>>> Meanwhile I'll try to dig to see if I am able to understand something more.
>>>>>>>>
>>>>>>>> Regards.
>>>>>>>>
>>>>>>>> [1] https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174
>>>>>>> [2] https://paste.apache.org/gxcm6
>>>>>>> [3] https://paste.apache.org/1g9k5
>>>>>>>
>>>>>>>> On 03/07/23 12:15, Mark Struberg wrote:
>>>>>>>>> Update: I've now merged the wip to our master and pushed it to the ASF gitox repo.
>>>>>>>>>
>>>>>>>>> I think we already came pretty far and it looks really promising. So there is a high chance that we succeed, although there is still plenty of work in front of us.
>>>>>>>>>
>>>>>>>>> How it works:
>>>>>>>>>
>>>>>>>>> Basically there are 2 sets of information in the PCEnhancer right now:
>>>>>>>>>
>>>>>>>>> _managed: the BCClass version of the original class
>>>>>>>>> _pc: the BCClass of the generated/modified class. Might be the same as _managed but in case of subclassing or interface case it is different. In those cases _managed represents the original class and _pc the subclass/concrete implementation.
>>>>>>>>>
>>>>>>>>> managed and pc are the ASM variants of _managed and _pc.
>>>>>>>>>
>>>>>>>>> The main entry point for debugging into it is PCEnhancer#run()
>>>>>>>>>
>>>>>>>>> There are a few methods in ASMHelper to update from BCClass -> ASM and vice versa:
>>>>>>>>>
>>>>>>>>> AsmHelper.readIntoBCClass(pc, _pc);
>>>>>>>>> takes all the information from the ASM pc, creates a class byte[] and reads that back into the existing _pc instance.
>>>>>>>>>
>>>>>>>>> pc = AsmHelper.toClassNode(_pc);
>>>>>>>>> is the corresponding method in the other direction. from BCClass to ASM.
>>>>>>>>>
>>>>>>>>> Right now we have the following methods left to migrate in PCEnhancer
>>>>>>>>> addAttachDetachCode();
>>>>>>>>> addSerializationCode();
>>>>>>>>> addCloningCode();
>>>>>>>>> runAuxiliaryEnhancers();
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'll be off for a week on holidays in Venice.
>>>>>>>>> If someone want's to join in with the effort then please go on! You might want to compare the current PCEnhancer code with an older version to get an idea how it can be done and to see some patterns I've used.
>>>>>>>>>
>>>>>>>>> As a quick hint: Java is basically a stack based virtual CPU. The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that comes the parameters. The size of the param on the stack depends on the type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
>>>>>>>>> In case of a static method there is ofc no 'this' on the stack, so the parameters start with zero offset.
>>>>>>>>> After all the parameters there are the unnamed 'local' variables. I tried to consistently name them xxxVarPos.
>>>>>>>>> See also the various helper methods in ASMHelper. For example for the various LOAD, RETURN, STORE operations which depends based on the type (ALOAD vs ILOAD vs IALOAD etc)
>>>>>>>>> Also read the guide to ASM PDF from objectweb.
>>>>>>>>>
>>>>>>>>> I'll be mostly offline till Sunday, but will try to answer mails if there are some questions.
>>>>>>>>>
>>>>>>>>> txs and LieGrue,
>>>>>>>>> strub
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <ma...@gmail.com>:
>>>>>>>>>>
>>>>>>>>>> +1 thanks for tackling this, Mark!
>>>>>>>>>>
>>>>>>>>>> ASM definitely more widely used going forward.
>>>>>>>>>>
>>>>>>>>>> -Matt
>>>>>>>>>>
>>>>>>>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg <st...@yahoo.de.INVALID> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Small update:
>>>>>>>>>>>
>>>>>>>>>>> I've worked on it over the last few weeks, and I'm getting closer
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>>>>
>>>>>>>>>>> contains the latest work on the PCEnhancer. Right in the middle of replacing Serp with native ASM code.
>>>>>>>>>>>
>>>>>>>>>>> Note that I also had to modify a few interfaces with a few more to follow from BCClass to ASM ClassNode.
>>>>>>>>>>>
>>>>>>>>>>> Happy to get some feedback!
>>>>>>>>>>>
>>>>>>>>>>> If anybody wants to help with this effort I'm happy to also push this feature branch to our ASF repo. It looks reasonably promising already.
>>>>>>>>>>>
>>>>>>>>>>> LieGrue,
>>>>>>>>>>> strub
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi!
>>>>>>>>>>>>
>>>>>>>>>>>> Right now I'm trying to get rid of serp step by step.
>>>>>>>>>>>> The code is right now at my own github repo in the fb_asmEnhance branch:
>>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The approach I took for now is to create a few methods in AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other way around. That way we should be able to replace functionality part by part but still keep all things afloat.
>>>>>>>>>>>>
>>>>>>>>>>>> For now I've started with the PCSubclassValidator.
>>>>>>>>>>>> Right now this evaluates the attributs using Serp plus ASM and then compare the results.
>>>>>>>>>>>> If something is fishy, you'll see the following
>>>>>>>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + " " + field);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It would be extremely helpful, if you could go through your projects and let it run and report how it's going.
>>>>>>>>>>>> If you see that "MSX ASMTODO" somewhere then we know I messed something up.
>>>>>>>>>>>>
>>>>>>>>>>>> I'll gonna finally remove the BCClass handling from those parts in a few days. Current commit is
>>>>>>>>>>>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>>>>>>>>>>>>
>>>>>>>>>>>> txs and LieGrue,
>>>>>>>>>>>> strub
>>
>
> --
> Francesco Chicchiriccò
>
> Tirasa - Open Source Excellence
> http://www.tirasa.net/
>
> Member at The Apache Software Foundation
> Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
> http://home.apache.org/~ilgrosso/
>
Re: enhance without Serp - Update
Posted by Francesco Chicchiriccò <il...@apache.org>.
Adding some info.
First of all, the class to which the error message is referring is not always the same.
Secondly, at least about AccountPolicy as reported below, the original error message is stating
Execution enhancer of goal org.apache.openjpa:openjpa-maven-plugin:4.0.0-SNAPSHOT:enhance failed: An error occurred while enhancing org.apache.syncope.core.persistence.jpa.entity.JPARealm. Exception message: Type org/apache/syncope/core/persistence/api/entity/policy/AccountPolicy not present: org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
If you look at JPARealm:
https://github.com/apache/syncope/blob/master/core/persistence-jpa/src/main/java/org/apache/syncope/core/persistence/jpa/entity/JPARealm.java#L82
it declares
@ManyToOne(fetch = FetchType.EAGER)
private JPAAccountPolicy accountPolicy;
e.g. a reference to the JPA entity JPAAccountPolicy, not to the interface that this class implements, e.g. AccountPolicy
It looks to me that somewhere the OpenJPA enhancement process is trying to resolve more than it should do.
Does this make ringing any bell?
Regards.
On 03/07/23 16:32, Francesco Chicchiriccò wrote:
> On 03/07/23 16:27, Mark Struberg wrote:
>> Went forward and compiled the Syncope main branch.
>>
>> Getting this error when running with mvn -X
>>
>> Caused by: java.lang.ClassNotFoundException: org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
>> at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass (SelfFirstStrategy.java:50)
>> at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass (ClassRealm.java:271)
>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:247)
>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:239)
>> at java.lang.Class.forName0 (Native Method)
>> at java.lang.Class.forName (Class.java:467)
>> at org.apache.xbean.asm9.ClassWriter.getCommonSuperClass (ClassWriter.java:1043)
>> at org.apache.xbean.asm9.SymbolTable.addMergedType (SymbolTable.java:1202)
>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1299)
>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1244)
>> at org.apache.xbean.asm9.MethodWriter.computeAllFrames (MethodWriter.java:1611)
>> at org.apache.xbean.asm9.MethodWriter.visitMaxs (MethodWriter.java:1547)
>> at org.apache.xbean.asm9.tree.MethodNode.accept (MethodNode.java:767)
>> at org.apache.xbean.asm9.tree.MethodNode.accept (MethodNode.java:647)
>> at org.apache.xbean.asm9.tree.ClassNode.accept (ClassNode.java:468)
>> at org.apache.openjpa.util.asm.AsmHelper.readIntoBCClass (AsmHelper.java:146)
>> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:606)
>> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:5889)
>> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:5829)
>> at org.apache.openjpa.enhance.PCEnhancer$1.run (PCEnhancer.java:5799)
>> at org.apache.openjpa.lib.conf.Configurations.launchRunnable (Configurations.java:760)
>> at org.apache.openjpa.lib.conf.Configurations.runAgainstAllAnchors (Configurations.java:745)
>>
>>
>> Is this the one you experience too?
>
> Yes, exactly.
>
> As you might have found out, AccountPolicy is an interface implemented by JPAAccountPolicy, which is the corresponding JPA entity.
> We take this approach for all JPA entities, actually.
>
> I thought the reason why AccountPolicy or JPAAccountPolicy are not found is that enhancement does not actually finds them.
>
>> Will take a look at it now.
>
> Thank you very much!
>
> Regards.
>>> Am 03.07.2023 um 15:49 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>>
>>> Hi Francesco!
>>>
>>> Can you see which one of the classes fail to enhance?
>>>
>>> In which case you might try to run the enhancement of that very class manually in your IDE.
>>>
>>> Main class: org.apache.openjpa.enhance.PCEnhancer
>>> VMOptions:: -ea -Dopenjpa.ConnectionURL=jdbc:derby:target/database/openjpa-derby-database;create=true -Dopenjpa.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver
>>> or something similar for your connection
>>>
>>> Then you also need the parameter pointing to the class file which you want to enhance, e.g.
>>> target/test-classes/org/apache/openjpa/persistence/proxy/entities/Annuity.class
>>>
>>> Please keep me updated!
>>>
>>>
>>> LieGrue,
>>> strub
>>>
>>> PS: does it add value if we'd write all this up? You will likely know most of that stuff, but others who dig into OpenJPA do not.
>>>
>>>
>>>
>>>> Am 03.07.2023 um 15:28 schrieb Francesco Chicchiriccò <il...@apache.org>:
>>>>
>>>> Hi Mark,
>>>> thanks for your reply.
>>>>
>>>> So I am proceeding as follows.
>>>>
>>>> Took the executions at
>>>>
>>>> https://ci-builds.apache.org/job/OpenJPA/job/OpenJPA-master-deploy/
>>>>
>>>> (the Jenkins job which is publishing SNAPSHOT artifacts from master to ASF Maven repo) and saw that the artifacts with which the Syncope build succeeded last Friday are the one from May 16th, so the corresponding commit id is
>>>>
>>>> b238417dabeac935846fe1ce2fef28aafaeef205
>>>>
>>>> I confirm that building locally from that commit id everything is working.
>>>>
>>>> As soon as I move to next commit (dated May 22nd, pushed and deployed today) with id
>>>>
>>>> b51d003ed98aa7663465aad7c9a1b554b006cc7d
>>>>
>>>> I see the same behavior as with HEAD, e.g.
>>>>
>>>> 2b9b024f273d63e479a02cad751c28b8ef974ace
>>>>
>>>> I will try to take a closer look at the code.
>>>> Regards.
>>>>
>>>> On 03/07/23 14:44, Mark Struberg wrote:
>>>>> Hi Francesco!
>>>>>
>>>>> That's very valuable information!
>>>>> Could you please do a git-bisect to find out which change did break it?
>>>>>
>>>>> Actually the algorithm to select which classes are to be enhanced should not have changed. Thus I'm really curious what I did break.
>>>>>
>>>>> Most times the amount of enhanced classes are the same but slightly differ in the generated code. That might be due to now we really generate Java11++ code whereas before we only generated Java 1.1 code (yes, not even 1.5!)
>>>>>
>>>>> What I usually do is to have 2 checkouts. You might want to change the pom GAV version of the working checkout to something something different so you might have both in your ~/.m2/repository in parallel.
>>>>>
>>>>> The first thing I do is to compare the class files in target/classes of both versions with 'compare with clipboard'.
>>>>> I just open the class file in target/classes and let Intellij/Netbeans do the decompilation for me.
>>>>>
>>>>> You might find some differences here in which case this needs further investigation. Note that the order of the methods might have changed.
>>>>> Sometimes you will also see some 'cannot decompile' in the code block. This happens when something in the bytecode is messed up. You can also watch out for ClassNotFound or VerifyError, etc.
>>>>>
>>>>> If this is the case I then use
>>>>> $> javap -c -p -constants target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile
>>>>> on both versions and check whether something is different.
>>>>> Often it's a wrong offset in the stack calculation (e.g. ALOAD 1 vs ALOAD 2)
>>>>>
>>>>> Note that the 'old' Java 1.1 bytecode doesn't understand LDC of Class constants (LoaD Constant). This was only introduced in Java 1.4 or 1.5 bytecode afair. Before that a dynamic Class.forName was called and stored in a static variable. Ugly stuff, and happy to finally get rid of all those hacks...
>>>>>
>>>>> Hope that helps to get you started. Feel free to ping me again over here or at the #apachee ASF slack channel.
>>>>>
>>>>> txs and LieGrue,
>>>>> strub
>>>>>
>>>>>
>>>>>
>>>>>> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò <il...@apache.org>:
>>>>>>
>>>>>> From a very preliminary analysis it seems that just a subset of the persistence-capable classes are effectively found and enhanced: now [2], it used to be [3].
>>>>>>
>>>>>> I'll continue investigating.
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>> On 03/07/23 13:56, Francesco Chicchiriccò wrote:
>>>>>>> Hi Mark,
>>>>>>> the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT is now failing - it used to work at least until last June 30th.
>>>>>>>
>>>>>>> The issues seem to be related to entity enhancement, which we perform via openjpa-maven-plugin - see [1].
>>>>>>>
>>>>>>> Is there anything obvious we should be changing?
>>>>>>> Meanwhile I'll try to dig to see if I am able to understand something more.
>>>>>>>
>>>>>>> Regards.
>>>>>>>
>>>>>>> [1] https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174
>>>>>> [2] https://paste.apache.org/gxcm6
>>>>>> [3] https://paste.apache.org/1g9k5
>>>>>>
>>>>>>> On 03/07/23 12:15, Mark Struberg wrote:
>>>>>>>> Update: I've now merged the wip to our master and pushed it to the ASF gitox repo.
>>>>>>>>
>>>>>>>> I think we already came pretty far and it looks really promising. So there is a high chance that we succeed, although there is still plenty of work in front of us.
>>>>>>>>
>>>>>>>> How it works:
>>>>>>>>
>>>>>>>> Basically there are 2 sets of information in the PCEnhancer right now:
>>>>>>>>
>>>>>>>> _managed: the BCClass version of the original class
>>>>>>>> _pc: the BCClass of the generated/modified class. Might be the same as _managed but in case of subclassing or interface case it is different. In those cases _managed represents the original class and _pc the subclass/concrete implementation.
>>>>>>>>
>>>>>>>> managed and pc are the ASM variants of _managed and _pc.
>>>>>>>>
>>>>>>>> The main entry point for debugging into it is PCEnhancer#run()
>>>>>>>>
>>>>>>>> There are a few methods in ASMHelper to update from BCClass -> ASM and vice versa:
>>>>>>>>
>>>>>>>> AsmHelper.readIntoBCClass(pc, _pc);
>>>>>>>> takes all the information from the ASM pc, creates a class byte[] and reads that back into the existing _pc instance.
>>>>>>>>
>>>>>>>> pc = AsmHelper.toClassNode(_pc);
>>>>>>>> is the corresponding method in the other direction. from BCClass to ASM.
>>>>>>>>
>>>>>>>> Right now we have the following methods left to migrate in PCEnhancer
>>>>>>>> addAttachDetachCode();
>>>>>>>> addSerializationCode();
>>>>>>>> addCloningCode();
>>>>>>>> runAuxiliaryEnhancers();
>>>>>>>>
>>>>>>>>
>>>>>>>> I'll be off for a week on holidays in Venice.
>>>>>>>> If someone want's to join in with the effort then please go on! You might want to compare the current PCEnhancer code with an older version to get an idea how it can be done and to see some patterns I've used.
>>>>>>>>
>>>>>>>> As a quick hint: Java is basically a stack based virtual CPU. The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that comes the parameters. The size of the param on the stack depends on the type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
>>>>>>>> In case of a static method there is ofc no 'this' on the stack, so the parameters start with zero offset.
>>>>>>>> After all the parameters there are the unnamed 'local' variables. I tried to consistently name them xxxVarPos.
>>>>>>>> See also the various helper methods in ASMHelper. For example for the various LOAD, RETURN, STORE operations which depends based on the type (ALOAD vs ILOAD vs IALOAD etc)
>>>>>>>> Also read the guide to ASM PDF from objectweb.
>>>>>>>>
>>>>>>>> I'll be mostly offline till Sunday, but will try to answer mails if there are some questions.
>>>>>>>>
>>>>>>>> txs and LieGrue,
>>>>>>>> strub
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <ma...@gmail.com>:
>>>>>>>>>
>>>>>>>>> +1 thanks for tackling this, Mark!
>>>>>>>>>
>>>>>>>>> ASM definitely more widely used going forward.
>>>>>>>>>
>>>>>>>>> -Matt
>>>>>>>>>
>>>>>>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg <st...@yahoo.de.INVALID> wrote:
>>>>>>>>>>
>>>>>>>>>> Small update:
>>>>>>>>>>
>>>>>>>>>> I've worked on it over the last few weeks, and I'm getting closer
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>>>
>>>>>>>>>> contains the latest work on the PCEnhancer. Right in the middle of replacing Serp with native ASM code.
>>>>>>>>>>
>>>>>>>>>> Note that I also had to modify a few interfaces with a few more to follow from BCClass to ASM ClassNode.
>>>>>>>>>>
>>>>>>>>>> Happy to get some feedback!
>>>>>>>>>>
>>>>>>>>>> If anybody wants to help with this effort I'm happy to also push this feature branch to our ASF repo. It looks reasonably promising already.
>>>>>>>>>>
>>>>>>>>>> LieGrue,
>>>>>>>>>> strub
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>>>>>>>>>>
>>>>>>>>>>> Hi!
>>>>>>>>>>>
>>>>>>>>>>> Right now I'm trying to get rid of serp step by step.
>>>>>>>>>>> The code is right now at my own github repo in the fb_asmEnhance branch:
>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The approach I took for now is to create a few methods in AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other way around. That way we should be able to replace functionality part by part but still keep all things afloat.
>>>>>>>>>>>
>>>>>>>>>>> For now I've started with the PCSubclassValidator.
>>>>>>>>>>> Right now this evaluates the attributs using Serp plus ASM and then compare the results.
>>>>>>>>>>> If something is fishy, you'll see the following
>>>>>>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + " " + field);
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It would be extremely helpful, if you could go through your projects and let it run and report how it's going.
>>>>>>>>>>> If you see that "MSX ASMTODO" somewhere then we know I messed something up.
>>>>>>>>>>>
>>>>>>>>>>> I'll gonna finally remove the BCClass handling from those parts in a few days. Current commit is
>>>>>>>>>>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>>>>>>>>>>>
>>>>>>>>>>> txs and LieGrue,
>>>>>>>>>>> strub
>
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Re: enhance without Serp - Update
Posted by Francesco Chicchiriccò <il...@apache.org>.
On 03/07/23 16:27, Mark Struberg wrote:
> Went forward and compiled the Syncope main branch.
>
> Getting this error when running with mvn -X
>
> Caused by: java.lang.ClassNotFoundException: org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
> at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass (SelfFirstStrategy.java:50)
> at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass (ClassRealm.java:271)
> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:247)
> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:239)
> at java.lang.Class.forName0 (Native Method)
> at java.lang.Class.forName (Class.java:467)
> at org.apache.xbean.asm9.ClassWriter.getCommonSuperClass (ClassWriter.java:1043)
> at org.apache.xbean.asm9.SymbolTable.addMergedType (SymbolTable.java:1202)
> at org.apache.xbean.asm9.Frame.merge (Frame.java:1299)
> at org.apache.xbean.asm9.Frame.merge (Frame.java:1244)
> at org.apache.xbean.asm9.MethodWriter.computeAllFrames (MethodWriter.java:1611)
> at org.apache.xbean.asm9.MethodWriter.visitMaxs (MethodWriter.java:1547)
> at org.apache.xbean.asm9.tree.MethodNode.accept (MethodNode.java:767)
> at org.apache.xbean.asm9.tree.MethodNode.accept (MethodNode.java:647)
> at org.apache.xbean.asm9.tree.ClassNode.accept (ClassNode.java:468)
> at org.apache.openjpa.util.asm.AsmHelper.readIntoBCClass (AsmHelper.java:146)
> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:606)
> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:5889)
> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:5829)
> at org.apache.openjpa.enhance.PCEnhancer$1.run (PCEnhancer.java:5799)
> at org.apache.openjpa.lib.conf.Configurations.launchRunnable (Configurations.java:760)
> at org.apache.openjpa.lib.conf.Configurations.runAgainstAllAnchors (Configurations.java:745)
>
>
> Is this the one you experience too?
Yes, exactly.
As you might have found out, AccountPolicy is an interface implemented by JPAAccountPolicy, which is the corresponding JPA entity.
We take this approach for all JPA entities, actually.
I thought the reason why AccountPolicy or JPAAccountPolicy are not found is that enhancement does not actually finds them.
> Will take a look at it now.
Thank you very much!
Regards.
>> Am 03.07.2023 um 15:49 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>
>> Hi Francesco!
>>
>> Can you see which one of the classes fail to enhance?
>>
>> In which case you might try to run the enhancement of that very class manually in your IDE.
>>
>> Main class: org.apache.openjpa.enhance.PCEnhancer
>> VMOptions:: -ea -Dopenjpa.ConnectionURL=jdbc:derby:target/database/openjpa-derby-database;create=true -Dopenjpa.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver
>> or something similar for your connection
>>
>> Then you also need the parameter pointing to the class file which you want to enhance, e.g.
>> target/test-classes/org/apache/openjpa/persistence/proxy/entities/Annuity.class
>>
>> Please keep me updated!
>>
>>
>> LieGrue,
>> strub
>>
>> PS: does it add value if we'd write all this up? You will likely know most of that stuff, but others who dig into OpenJPA do not.
>>
>>
>>
>>> Am 03.07.2023 um 15:28 schrieb Francesco Chicchiriccò <il...@apache.org>:
>>>
>>> Hi Mark,
>>> thanks for your reply.
>>>
>>> So I am proceeding as follows.
>>>
>>> Took the executions at
>>>
>>> https://ci-builds.apache.org/job/OpenJPA/job/OpenJPA-master-deploy/
>>>
>>> (the Jenkins job which is publishing SNAPSHOT artifacts from master to ASF Maven repo) and saw that the artifacts with which the Syncope build succeeded last Friday are the one from May 16th, so the corresponding commit id is
>>>
>>> b238417dabeac935846fe1ce2fef28aafaeef205
>>>
>>> I confirm that building locally from that commit id everything is working.
>>>
>>> As soon as I move to next commit (dated May 22nd, pushed and deployed today) with id
>>>
>>> b51d003ed98aa7663465aad7c9a1b554b006cc7d
>>>
>>> I see the same behavior as with HEAD, e.g.
>>>
>>> 2b9b024f273d63e479a02cad751c28b8ef974ace
>>>
>>> I will try to take a closer look at the code.
>>> Regards.
>>>
>>> On 03/07/23 14:44, Mark Struberg wrote:
>>>> Hi Francesco!
>>>>
>>>> That's very valuable information!
>>>> Could you please do a git-bisect to find out which change did break it?
>>>>
>>>> Actually the algorithm to select which classes are to be enhanced should not have changed. Thus I'm really curious what I did break.
>>>>
>>>> Most times the amount of enhanced classes are the same but slightly differ in the generated code. That might be due to now we really generate Java11++ code whereas before we only generated Java 1.1 code (yes, not even 1.5!)
>>>>
>>>> What I usually do is to have 2 checkouts. You might want to change the pom GAV version of the working checkout to something something different so you might have both in your ~/.m2/repository in parallel.
>>>>
>>>> The first thing I do is to compare the class files in target/classes of both versions with 'compare with clipboard'.
>>>> I just open the class file in target/classes and let Intellij/Netbeans do the decompilation for me.
>>>>
>>>> You might find some differences here in which case this needs further investigation. Note that the order of the methods might have changed.
>>>> Sometimes you will also see some 'cannot decompile' in the code block. This happens when something in the bytecode is messed up. You can also watch out for ClassNotFound or VerifyError, etc.
>>>>
>>>> If this is the case I then use
>>>> $> javap -c -p -constants target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile
>>>> on both versions and check whether something is different.
>>>> Often it's a wrong offset in the stack calculation (e.g. ALOAD 1 vs ALOAD 2)
>>>>
>>>> Note that the 'old' Java 1.1 bytecode doesn't understand LDC of Class constants (LoaD Constant). This was only introduced in Java 1.4 or 1.5 bytecode afair. Before that a dynamic Class.forName was called and stored in a static variable. Ugly stuff, and happy to finally get rid of all those hacks...
>>>>
>>>> Hope that helps to get you started. Feel free to ping me again over here or at the #apachee ASF slack channel.
>>>>
>>>> txs and LieGrue,
>>>> strub
>>>>
>>>>
>>>>
>>>>> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò <il...@apache.org>:
>>>>>
>>>>> From a very preliminary analysis it seems that just a subset of the persistence-capable classes are effectively found and enhanced: now [2], it used to be [3].
>>>>>
>>>>> I'll continue investigating.
>>>>>
>>>>> Regards.
>>>>>
>>>>> On 03/07/23 13:56, Francesco Chicchiriccò wrote:
>>>>>> Hi Mark,
>>>>>> the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT is now failing - it used to work at least until last June 30th.
>>>>>>
>>>>>> The issues seem to be related to entity enhancement, which we perform via openjpa-maven-plugin - see [1].
>>>>>>
>>>>>> Is there anything obvious we should be changing?
>>>>>> Meanwhile I'll try to dig to see if I am able to understand something more.
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>> [1] https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174
>>>>> [2] https://paste.apache.org/gxcm6
>>>>> [3] https://paste.apache.org/1g9k5
>>>>>
>>>>>> On 03/07/23 12:15, Mark Struberg wrote:
>>>>>>> Update: I've now merged the wip to our master and pushed it to the ASF gitox repo.
>>>>>>>
>>>>>>> I think we already came pretty far and it looks really promising. So there is a high chance that we succeed, although there is still plenty of work in front of us.
>>>>>>>
>>>>>>> How it works:
>>>>>>>
>>>>>>> Basically there are 2 sets of information in the PCEnhancer right now:
>>>>>>>
>>>>>>> _managed: the BCClass version of the original class
>>>>>>> _pc: the BCClass of the generated/modified class. Might be the same as _managed but in case of subclassing or interface case it is different. In those cases _managed represents the original class and _pc the subclass/concrete implementation.
>>>>>>>
>>>>>>> managed and pc are the ASM variants of _managed and _pc.
>>>>>>>
>>>>>>> The main entry point for debugging into it is PCEnhancer#run()
>>>>>>>
>>>>>>> There are a few methods in ASMHelper to update from BCClass -> ASM and vice versa:
>>>>>>>
>>>>>>> AsmHelper.readIntoBCClass(pc, _pc);
>>>>>>> takes all the information from the ASM pc, creates a class byte[] and reads that back into the existing _pc instance.
>>>>>>>
>>>>>>> pc = AsmHelper.toClassNode(_pc);
>>>>>>> is the corresponding method in the other direction. from BCClass to ASM.
>>>>>>>
>>>>>>> Right now we have the following methods left to migrate in PCEnhancer
>>>>>>> addAttachDetachCode();
>>>>>>> addSerializationCode();
>>>>>>> addCloningCode();
>>>>>>> runAuxiliaryEnhancers();
>>>>>>>
>>>>>>>
>>>>>>> I'll be off for a week on holidays in Venice.
>>>>>>> If someone want's to join in with the effort then please go on! You might want to compare the current PCEnhancer code with an older version to get an idea how it can be done and to see some patterns I've used.
>>>>>>>
>>>>>>> As a quick hint: Java is basically a stack based virtual CPU. The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that comes the parameters. The size of the param on the stack depends on the type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
>>>>>>> In case of a static method there is ofc no 'this' on the stack, so the parameters start with zero offset.
>>>>>>> After all the parameters there are the unnamed 'local' variables. I tried to consistently name them xxxVarPos.
>>>>>>> See also the various helper methods in ASMHelper. For example for the various LOAD, RETURN, STORE operations which depends based on the type (ALOAD vs ILOAD vs IALOAD etc)
>>>>>>> Also read the guide to ASM PDF from objectweb.
>>>>>>>
>>>>>>> I'll be mostly offline till Sunday, but will try to answer mails if there are some questions.
>>>>>>>
>>>>>>> txs and LieGrue,
>>>>>>> strub
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <ma...@gmail.com>:
>>>>>>>>
>>>>>>>> +1 thanks for tackling this, Mark!
>>>>>>>>
>>>>>>>> ASM definitely more widely used going forward.
>>>>>>>>
>>>>>>>> -Matt
>>>>>>>>
>>>>>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg <st...@yahoo.de.INVALID> wrote:
>>>>>>>>>
>>>>>>>>> Small update:
>>>>>>>>>
>>>>>>>>> I've worked on it over the last few weeks, and I'm getting closer
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>>
>>>>>>>>> contains the latest work on the PCEnhancer. Right in the middle of replacing Serp with native ASM code.
>>>>>>>>>
>>>>>>>>> Note that I also had to modify a few interfaces with a few more to follow from BCClass to ASM ClassNode.
>>>>>>>>>
>>>>>>>>> Happy to get some feedback!
>>>>>>>>>
>>>>>>>>> If anybody wants to help with this effort I'm happy to also push this feature branch to our ASF repo. It looks reasonably promising already.
>>>>>>>>>
>>>>>>>>> LieGrue,
>>>>>>>>> strub
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>>>>>>>>>
>>>>>>>>>> Hi!
>>>>>>>>>>
>>>>>>>>>> Right now I'm trying to get rid of serp step by step.
>>>>>>>>>> The code is right now at my own github repo in the fb_asmEnhance branch:
>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The approach I took for now is to create a few methods in AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other way around. That way we should be able to replace functionality part by part but still keep all things afloat.
>>>>>>>>>>
>>>>>>>>>> For now I've started with the PCSubclassValidator.
>>>>>>>>>> Right now this evaluates the attributs using Serp plus ASM and then compare the results.
>>>>>>>>>> If something is fishy, you'll see the following
>>>>>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + " " + field);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It would be extremely helpful, if you could go through your projects and let it run and report how it's going.
>>>>>>>>>> If you see that "MSX ASMTODO" somewhere then we know I messed something up.
>>>>>>>>>>
>>>>>>>>>> I'll gonna finally remove the BCClass handling from those parts in a few days. Current commit is
>>>>>>>>>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>>>>>>>>>>
>>>>>>>>>> txs and LieGrue,
>>>>>>>>>> strub
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Re: enhance without Serp - Update
Posted by Mark Struberg <st...@yahoo.de.INVALID>.
Went forward and compiled the Syncope main branch.
Getting this error when running with mvn -X
Caused by: java.lang.ClassNotFoundException: org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy
at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass (SelfFirstStrategy.java:50)
at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass (ClassRealm.java:271)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:247)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:239)
at java.lang.Class.forName0 (Native Method)
at java.lang.Class.forName (Class.java:467)
at org.apache.xbean.asm9.ClassWriter.getCommonSuperClass (ClassWriter.java:1043)
at org.apache.xbean.asm9.SymbolTable.addMergedType (SymbolTable.java:1202)
at org.apache.xbean.asm9.Frame.merge (Frame.java:1299)
at org.apache.xbean.asm9.Frame.merge (Frame.java:1244)
at org.apache.xbean.asm9.MethodWriter.computeAllFrames (MethodWriter.java:1611)
at org.apache.xbean.asm9.MethodWriter.visitMaxs (MethodWriter.java:1547)
at org.apache.xbean.asm9.tree.MethodNode.accept (MethodNode.java:767)
at org.apache.xbean.asm9.tree.MethodNode.accept (MethodNode.java:647)
at org.apache.xbean.asm9.tree.ClassNode.accept (ClassNode.java:468)
at org.apache.openjpa.util.asm.AsmHelper.readIntoBCClass (AsmHelper.java:146)
at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:606)
at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:5889)
at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:5829)
at org.apache.openjpa.enhance.PCEnhancer$1.run (PCEnhancer.java:5799)
at org.apache.openjpa.lib.conf.Configurations.launchRunnable (Configurations.java:760)
at org.apache.openjpa.lib.conf.Configurations.runAgainstAllAnchors (Configurations.java:745)
Is this the one you experience too?
Will take a look at it now.
LieGrue
strub
> Am 03.07.2023 um 15:49 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>
> Hi Francesco!
>
> Can you see which one of the classes fail to enhance?
>
> In which case you might try to run the enhancement of that very class manually in your IDE.
>
> Main class: org.apache.openjpa.enhance.PCEnhancer
> VMOptions:: -ea -Dopenjpa.ConnectionURL=jdbc:derby:target/database/openjpa-derby-database;create=true -Dopenjpa.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver
> or something similar for your connection
>
> Then you also need the parameter pointing to the class file which you want to enhance, e.g.
> target/test-classes/org/apache/openjpa/persistence/proxy/entities/Annuity.class
>
> Please keep me updated!
>
>
> LieGrue,
> strub
>
> PS: does it add value if we'd write all this up? You will likely know most of that stuff, but others who dig into OpenJPA do not.
>
>
>
>> Am 03.07.2023 um 15:28 schrieb Francesco Chicchiriccò <il...@apache.org>:
>>
>> Hi Mark,
>> thanks for your reply.
>>
>> So I am proceeding as follows.
>>
>> Took the executions at
>>
>> https://ci-builds.apache.org/job/OpenJPA/job/OpenJPA-master-deploy/
>>
>> (the Jenkins job which is publishing SNAPSHOT artifacts from master to ASF Maven repo) and saw that the artifacts with which the Syncope build succeeded last Friday are the one from May 16th, so the corresponding commit id is
>>
>> b238417dabeac935846fe1ce2fef28aafaeef205
>>
>> I confirm that building locally from that commit id everything is working.
>>
>> As soon as I move to next commit (dated May 22nd, pushed and deployed today) with id
>>
>> b51d003ed98aa7663465aad7c9a1b554b006cc7d
>>
>> I see the same behavior as with HEAD, e.g.
>>
>> 2b9b024f273d63e479a02cad751c28b8ef974ace
>>
>> I will try to take a closer look at the code.
>> Regards.
>>
>> On 03/07/23 14:44, Mark Struberg wrote:
>>> Hi Francesco!
>>>
>>> That's very valuable information!
>>> Could you please do a git-bisect to find out which change did break it?
>>>
>>> Actually the algorithm to select which classes are to be enhanced should not have changed. Thus I'm really curious what I did break.
>>>
>>> Most times the amount of enhanced classes are the same but slightly differ in the generated code. That might be due to now we really generate Java11++ code whereas before we only generated Java 1.1 code (yes, not even 1.5!)
>>>
>>> What I usually do is to have 2 checkouts. You might want to change the pom GAV version of the working checkout to something something different so you might have both in your ~/.m2/repository in parallel.
>>>
>>> The first thing I do is to compare the class files in target/classes of both versions with 'compare with clipboard'.
>>> I just open the class file in target/classes and let Intellij/Netbeans do the decompilation for me.
>>>
>>> You might find some differences here in which case this needs further investigation. Note that the order of the methods might have changed.
>>> Sometimes you will also see some 'cannot decompile' in the code block. This happens when something in the bytecode is messed up. You can also watch out for ClassNotFound or VerifyError, etc.
>>>
>>> If this is the case I then use
>>> $> javap -c -p -constants target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile
>>> on both versions and check whether something is different.
>>> Often it's a wrong offset in the stack calculation (e.g. ALOAD 1 vs ALOAD 2)
>>>
>>> Note that the 'old' Java 1.1 bytecode doesn't understand LDC of Class constants (LoaD Constant). This was only introduced in Java 1.4 or 1.5 bytecode afair. Before that a dynamic Class.forName was called and stored in a static variable. Ugly stuff, and happy to finally get rid of all those hacks...
>>>
>>> Hope that helps to get you started. Feel free to ping me again over here or at the #apachee ASF slack channel.
>>>
>>> txs and LieGrue,
>>> strub
>>>
>>>
>>>
>>>> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò <il...@apache.org>:
>>>>
>>>> From a very preliminary analysis it seems that just a subset of the persistence-capable classes are effectively found and enhanced: now [2], it used to be [3].
>>>>
>>>> I'll continue investigating.
>>>>
>>>> Regards.
>>>>
>>>> On 03/07/23 13:56, Francesco Chicchiriccò wrote:
>>>>> Hi Mark,
>>>>> the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT is now failing - it used to work at least until last June 30th.
>>>>>
>>>>> The issues seem to be related to entity enhancement, which we perform via openjpa-maven-plugin - see [1].
>>>>>
>>>>> Is there anything obvious we should be changing?
>>>>> Meanwhile I'll try to dig to see if I am able to understand something more.
>>>>>
>>>>> Regards.
>>>>>
>>>>> [1] https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174
>>>> [2] https://paste.apache.org/gxcm6
>>>> [3] https://paste.apache.org/1g9k5
>>>>
>>>>> On 03/07/23 12:15, Mark Struberg wrote:
>>>>>> Update: I've now merged the wip to our master and pushed it to the ASF gitox repo.
>>>>>>
>>>>>> I think we already came pretty far and it looks really promising. So there is a high chance that we succeed, although there is still plenty of work in front of us.
>>>>>>
>>>>>> How it works:
>>>>>>
>>>>>> Basically there are 2 sets of information in the PCEnhancer right now:
>>>>>>
>>>>>> _managed: the BCClass version of the original class
>>>>>> _pc: the BCClass of the generated/modified class. Might be the same as _managed but in case of subclassing or interface case it is different. In those cases _managed represents the original class and _pc the subclass/concrete implementation.
>>>>>>
>>>>>> managed and pc are the ASM variants of _managed and _pc.
>>>>>>
>>>>>> The main entry point for debugging into it is PCEnhancer#run()
>>>>>>
>>>>>> There are a few methods in ASMHelper to update from BCClass -> ASM and vice versa:
>>>>>>
>>>>>> AsmHelper.readIntoBCClass(pc, _pc);
>>>>>> takes all the information from the ASM pc, creates a class byte[] and reads that back into the existing _pc instance.
>>>>>>
>>>>>> pc = AsmHelper.toClassNode(_pc);
>>>>>> is the corresponding method in the other direction. from BCClass to ASM.
>>>>>>
>>>>>> Right now we have the following methods left to migrate in PCEnhancer
>>>>>> addAttachDetachCode();
>>>>>> addSerializationCode();
>>>>>> addCloningCode();
>>>>>> runAuxiliaryEnhancers();
>>>>>>
>>>>>>
>>>>>> I'll be off for a week on holidays in Venice.
>>>>>> If someone want's to join in with the effort then please go on! You might want to compare the current PCEnhancer code with an older version to get an idea how it can be done and to see some patterns I've used.
>>>>>>
>>>>>> As a quick hint: Java is basically a stack based virtual CPU. The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that comes the parameters. The size of the param on the stack depends on the type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
>>>>>> In case of a static method there is ofc no 'this' on the stack, so the parameters start with zero offset.
>>>>>> After all the parameters there are the unnamed 'local' variables. I tried to consistently name them xxxVarPos.
>>>>>> See also the various helper methods in ASMHelper. For example for the various LOAD, RETURN, STORE operations which depends based on the type (ALOAD vs ILOAD vs IALOAD etc)
>>>>>> Also read the guide to ASM PDF from objectweb.
>>>>>>
>>>>>> I'll be mostly offline till Sunday, but will try to answer mails if there are some questions.
>>>>>>
>>>>>> txs and LieGrue,
>>>>>> strub
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <ma...@gmail.com>:
>>>>>>>
>>>>>>> +1 thanks for tackling this, Mark!
>>>>>>>
>>>>>>> ASM definitely more widely used going forward.
>>>>>>>
>>>>>>> -Matt
>>>>>>>
>>>>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg <st...@yahoo.de.INVALID> wrote:
>>>>>>>>
>>>>>>>> Small update:
>>>>>>>>
>>>>>>>> I've worked on it over the last few weeks, and I'm getting closer
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>
>>>>>>>> contains the latest work on the PCEnhancer. Right in the middle of replacing Serp with native ASM code.
>>>>>>>>
>>>>>>>> Note that I also had to modify a few interfaces with a few more to follow from BCClass to ASM ClassNode.
>>>>>>>>
>>>>>>>> Happy to get some feedback!
>>>>>>>>
>>>>>>>> If anybody wants to help with this effort I'm happy to also push this feature branch to our ASF repo. It looks reasonably promising already.
>>>>>>>>
>>>>>>>> LieGrue,
>>>>>>>> strub
>>>>>>>>
>>>>>>>>
>>>>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>>>>>>>>
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>> Right now I'm trying to get rid of serp step by step.
>>>>>>>>> The code is right now at my own github repo in the fb_asmEnhance branch:
>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The approach I took for now is to create a few methods in AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other way around. That way we should be able to replace functionality part by part but still keep all things afloat.
>>>>>>>>>
>>>>>>>>> For now I've started with the PCSubclassValidator.
>>>>>>>>> Right now this evaluates the attributs using Serp plus ASM and then compare the results.
>>>>>>>>> If something is fishy, you'll see the following
>>>>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + " " + field);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It would be extremely helpful, if you could go through your projects and let it run and report how it's going.
>>>>>>>>> If you see that "MSX ASMTODO" somewhere then we know I messed something up.
>>>>>>>>>
>>>>>>>>> I'll gonna finally remove the BCClass handling from those parts in a few days. Current commit is
>>>>>>>>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>>>>>>>>>
>>>>>>>>> txs and LieGrue,
>>>>>>>>> strub
>>
>> --
>> Francesco Chicchiriccò
>>
>> Tirasa - Open Source Excellence
>> http://www.tirasa.net/
>>
>> Member at The Apache Software Foundation
>> Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
>> http://home.apache.org/~ilgrosso/
>>
>
Re: enhance without Serp - Update
Posted by Mark Struberg <st...@yahoo.de.INVALID>.
Hi Francesco!
Can you see which one of the classes fail to enhance?
In which case you might try to run the enhancement of that very class manually in your IDE.
Main class: org.apache.openjpa.enhance.PCEnhancer
VMOptions:: -ea -Dopenjpa.ConnectionURL=jdbc:derby:target/database/openjpa-derby-database;create=true -Dopenjpa.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver
or something similar for your connection
Then you also need the parameter pointing to the class file which you want to enhance, e.g.
target/test-classes/org/apache/openjpa/persistence/proxy/entities/Annuity.class
Please keep me updated!
LieGrue,
strub
PS: does it add value if we'd write all this up? You will likely know most of that stuff, but others who dig into OpenJPA do not.
> Am 03.07.2023 um 15:28 schrieb Francesco Chicchiriccò <il...@apache.org>:
>
> Hi Mark,
> thanks for your reply.
>
> So I am proceeding as follows.
>
> Took the executions at
>
> https://ci-builds.apache.org/job/OpenJPA/job/OpenJPA-master-deploy/
>
> (the Jenkins job which is publishing SNAPSHOT artifacts from master to ASF Maven repo) and saw that the artifacts with which the Syncope build succeeded last Friday are the one from May 16th, so the corresponding commit id is
>
> b238417dabeac935846fe1ce2fef28aafaeef205
>
> I confirm that building locally from that commit id everything is working.
>
> As soon as I move to next commit (dated May 22nd, pushed and deployed today) with id
>
> b51d003ed98aa7663465aad7c9a1b554b006cc7d
>
> I see the same behavior as with HEAD, e.g.
>
> 2b9b024f273d63e479a02cad751c28b8ef974ace
>
> I will try to take a closer look at the code.
> Regards.
>
> On 03/07/23 14:44, Mark Struberg wrote:
>> Hi Francesco!
>>
>> That's very valuable information!
>> Could you please do a git-bisect to find out which change did break it?
>>
>> Actually the algorithm to select which classes are to be enhanced should not have changed. Thus I'm really curious what I did break.
>>
>> Most times the amount of enhanced classes are the same but slightly differ in the generated code. That might be due to now we really generate Java11++ code whereas before we only generated Java 1.1 code (yes, not even 1.5!)
>>
>> What I usually do is to have 2 checkouts. You might want to change the pom GAV version of the working checkout to something something different so you might have both in your ~/.m2/repository in parallel.
>>
>> The first thing I do is to compare the class files in target/classes of both versions with 'compare with clipboard'.
>> I just open the class file in target/classes and let Intellij/Netbeans do the decompilation for me.
>>
>> You might find some differences here in which case this needs further investigation. Note that the order of the methods might have changed.
>> Sometimes you will also see some 'cannot decompile' in the code block. This happens when something in the bytecode is messed up. You can also watch out for ClassNotFound or VerifyError, etc.
>>
>> If this is the case I then use
>> $> javap -c -p -constants target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile
>> on both versions and check whether something is different.
>> Often it's a wrong offset in the stack calculation (e.g. ALOAD 1 vs ALOAD 2)
>>
>> Note that the 'old' Java 1.1 bytecode doesn't understand LDC of Class constants (LoaD Constant). This was only introduced in Java 1.4 or 1.5 bytecode afair. Before that a dynamic Class.forName was called and stored in a static variable. Ugly stuff, and happy to finally get rid of all those hacks...
>>
>> Hope that helps to get you started. Feel free to ping me again over here or at the #apachee ASF slack channel.
>>
>> txs and LieGrue,
>> strub
>>
>>
>>
>>> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò <il...@apache.org>:
>>>
>>> From a very preliminary analysis it seems that just a subset of the persistence-capable classes are effectively found and enhanced: now [2], it used to be [3].
>>>
>>> I'll continue investigating.
>>>
>>> Regards.
>>>
>>> On 03/07/23 13:56, Francesco Chicchiriccò wrote:
>>>> Hi Mark,
>>>> the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT is now failing - it used to work at least until last June 30th.
>>>>
>>>> The issues seem to be related to entity enhancement, which we perform via openjpa-maven-plugin - see [1].
>>>>
>>>> Is there anything obvious we should be changing?
>>>> Meanwhile I'll try to dig to see if I am able to understand something more.
>>>>
>>>> Regards.
>>>>
>>>> [1] https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174
>>> [2] https://paste.apache.org/gxcm6
>>> [3] https://paste.apache.org/1g9k5
>>>
>>>> On 03/07/23 12:15, Mark Struberg wrote:
>>>>> Update: I've now merged the wip to our master and pushed it to the ASF gitox repo.
>>>>>
>>>>> I think we already came pretty far and it looks really promising. So there is a high chance that we succeed, although there is still plenty of work in front of us.
>>>>>
>>>>> How it works:
>>>>>
>>>>> Basically there are 2 sets of information in the PCEnhancer right now:
>>>>>
>>>>> _managed: the BCClass version of the original class
>>>>> _pc: the BCClass of the generated/modified class. Might be the same as _managed but in case of subclassing or interface case it is different. In those cases _managed represents the original class and _pc the subclass/concrete implementation.
>>>>>
>>>>> managed and pc are the ASM variants of _managed and _pc.
>>>>>
>>>>> The main entry point for debugging into it is PCEnhancer#run()
>>>>>
>>>>> There are a few methods in ASMHelper to update from BCClass -> ASM and vice versa:
>>>>>
>>>>> AsmHelper.readIntoBCClass(pc, _pc);
>>>>> takes all the information from the ASM pc, creates a class byte[] and reads that back into the existing _pc instance.
>>>>>
>>>>> pc = AsmHelper.toClassNode(_pc);
>>>>> is the corresponding method in the other direction. from BCClass to ASM.
>>>>>
>>>>> Right now we have the following methods left to migrate in PCEnhancer
>>>>> addAttachDetachCode();
>>>>> addSerializationCode();
>>>>> addCloningCode();
>>>>> runAuxiliaryEnhancers();
>>>>>
>>>>>
>>>>> I'll be off for a week on holidays in Venice.
>>>>> If someone want's to join in with the effort then please go on! You might want to compare the current PCEnhancer code with an older version to get an idea how it can be done and to see some patterns I've used.
>>>>>
>>>>> As a quick hint: Java is basically a stack based virtual CPU. The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that comes the parameters. The size of the param on the stack depends on the type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
>>>>> In case of a static method there is ofc no 'this' on the stack, so the parameters start with zero offset.
>>>>> After all the parameters there are the unnamed 'local' variables. I tried to consistently name them xxxVarPos.
>>>>> See also the various helper methods in ASMHelper. For example for the various LOAD, RETURN, STORE operations which depends based on the type (ALOAD vs ILOAD vs IALOAD etc)
>>>>> Also read the guide to ASM PDF from objectweb.
>>>>>
>>>>> I'll be mostly offline till Sunday, but will try to answer mails if there are some questions.
>>>>>
>>>>> txs and LieGrue,
>>>>> strub
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <ma...@gmail.com>:
>>>>>>
>>>>>> +1 thanks for tackling this, Mark!
>>>>>>
>>>>>> ASM definitely more widely used going forward.
>>>>>>
>>>>>> -Matt
>>>>>>
>>>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg <st...@yahoo.de.INVALID> wrote:
>>>>>>>
>>>>>>> Small update:
>>>>>>>
>>>>>>> I've worked on it over the last few weeks, and I'm getting closer
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>
>>>>>>> contains the latest work on the PCEnhancer. Right in the middle of replacing Serp with native ASM code.
>>>>>>>
>>>>>>> Note that I also had to modify a few interfaces with a few more to follow from BCClass to ASM ClassNode.
>>>>>>>
>>>>>>> Happy to get some feedback!
>>>>>>>
>>>>>>> If anybody wants to help with this effort I'm happy to also push this feature branch to our ASF repo. It looks reasonably promising already.
>>>>>>>
>>>>>>> LieGrue,
>>>>>>> strub
>>>>>>>
>>>>>>>
>>>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> Right now I'm trying to get rid of serp step by step.
>>>>>>>> The code is right now at my own github repo in the fb_asmEnhance branch:
>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> The approach I took for now is to create a few methods in AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other way around. That way we should be able to replace functionality part by part but still keep all things afloat.
>>>>>>>>
>>>>>>>> For now I've started with the PCSubclassValidator.
>>>>>>>> Right now this evaluates the attributs using Serp plus ASM and then compare the results.
>>>>>>>> If something is fishy, you'll see the following
>>>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + " " + field);
>>>>>>>>
>>>>>>>>
>>>>>>>> It would be extremely helpful, if you could go through your projects and let it run and report how it's going.
>>>>>>>> If you see that "MSX ASMTODO" somewhere then we know I messed something up.
>>>>>>>>
>>>>>>>> I'll gonna finally remove the BCClass handling from those parts in a few days. Current commit is
>>>>>>>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>>>>>>>>
>>>>>>>> txs and LieGrue,
>>>>>>>> strub
>
> --
> Francesco Chicchiriccò
>
> Tirasa - Open Source Excellence
> http://www.tirasa.net/
>
> Member at The Apache Software Foundation
> Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
> http://home.apache.org/~ilgrosso/
>
Re: enhance without Serp - Update
Posted by Francesco Chicchiriccò <il...@apache.org>.
Hi Mark,
thanks for your reply.
So I am proceeding as follows.
Took the executions at
https://ci-builds.apache.org/job/OpenJPA/job/OpenJPA-master-deploy/
(the Jenkins job which is publishing SNAPSHOT artifacts from master to ASF Maven repo) and saw that the artifacts with which the Syncope build succeeded last Friday are the one from May 16th, so the corresponding commit id is
b238417dabeac935846fe1ce2fef28aafaeef205
I confirm that building locally from that commit id everything is working.
As soon as I move to next commit (dated May 22nd, pushed and deployed today) with id
b51d003ed98aa7663465aad7c9a1b554b006cc7d
I see the same behavior as with HEAD, e.g.
2b9b024f273d63e479a02cad751c28b8ef974ace
I will try to take a closer look at the code.
Regards.
On 03/07/23 14:44, Mark Struberg wrote:
> Hi Francesco!
>
> That's very valuable information!
> Could you please do a git-bisect to find out which change did break it?
>
> Actually the algorithm to select which classes are to be enhanced should not have changed. Thus I'm really curious what I did break.
>
> Most times the amount of enhanced classes are the same but slightly differ in the generated code. That might be due to now we really generate Java11++ code whereas before we only generated Java 1.1 code (yes, not even 1.5!)
>
> What I usually do is to have 2 checkouts. You might want to change the pom GAV version of the working checkout to something something different so you might have both in your ~/.m2/repository in parallel.
>
> The first thing I do is to compare the class files in target/classes of both versions with 'compare with clipboard'.
> I just open the class file in target/classes and let Intellij/Netbeans do the decompilation for me.
>
> You might find some differences here in which case this needs further investigation. Note that the order of the methods might have changed.
> Sometimes you will also see some 'cannot decompile' in the code block. This happens when something in the bytecode is messed up. You can also watch out for ClassNotFound or VerifyError, etc.
>
> If this is the case I then use
> $> javap -c -p -constants target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile
> on both versions and check whether something is different.
> Often it's a wrong offset in the stack calculation (e.g. ALOAD 1 vs ALOAD 2)
>
> Note that the 'old' Java 1.1 bytecode doesn't understand LDC of Class constants (LoaD Constant). This was only introduced in Java 1.4 or 1.5 bytecode afair. Before that a dynamic Class.forName was called and stored in a static variable. Ugly stuff, and happy to finally get rid of all those hacks...
>
> Hope that helps to get you started. Feel free to ping me again over here or at the #apachee ASF slack channel.
>
> txs and LieGrue,
> strub
>
>
>
>> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò <il...@apache.org>:
>>
>> From a very preliminary analysis it seems that just a subset of the persistence-capable classes are effectively found and enhanced: now [2], it used to be [3].
>>
>> I'll continue investigating.
>>
>> Regards.
>>
>> On 03/07/23 13:56, Francesco Chicchiriccò wrote:
>>> Hi Mark,
>>> the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT is now failing - it used to work at least until last June 30th.
>>>
>>> The issues seem to be related to entity enhancement, which we perform via openjpa-maven-plugin - see [1].
>>>
>>> Is there anything obvious we should be changing?
>>> Meanwhile I'll try to dig to see if I am able to understand something more.
>>>
>>> Regards.
>>>
>>> [1] https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174
>> [2] https://paste.apache.org/gxcm6
>> [3] https://paste.apache.org/1g9k5
>>
>>> On 03/07/23 12:15, Mark Struberg wrote:
>>>> Update: I've now merged the wip to our master and pushed it to the ASF gitox repo.
>>>>
>>>> I think we already came pretty far and it looks really promising. So there is a high chance that we succeed, although there is still plenty of work in front of us.
>>>>
>>>> How it works:
>>>>
>>>> Basically there are 2 sets of information in the PCEnhancer right now:
>>>>
>>>> _managed: the BCClass version of the original class
>>>> _pc: the BCClass of the generated/modified class. Might be the same as _managed but in case of subclassing or interface case it is different. In those cases _managed represents the original class and _pc the subclass/concrete implementation.
>>>>
>>>> managed and pc are the ASM variants of _managed and _pc.
>>>>
>>>> The main entry point for debugging into it is PCEnhancer#run()
>>>>
>>>> There are a few methods in ASMHelper to update from BCClass -> ASM and vice versa:
>>>>
>>>> AsmHelper.readIntoBCClass(pc, _pc);
>>>> takes all the information from the ASM pc, creates a class byte[] and reads that back into the existing _pc instance.
>>>>
>>>> pc = AsmHelper.toClassNode(_pc);
>>>> is the corresponding method in the other direction. from BCClass to ASM.
>>>>
>>>> Right now we have the following methods left to migrate in PCEnhancer
>>>> addAttachDetachCode();
>>>> addSerializationCode();
>>>> addCloningCode();
>>>> runAuxiliaryEnhancers();
>>>>
>>>>
>>>> I'll be off for a week on holidays in Venice.
>>>> If someone want's to join in with the effort then please go on! You might want to compare the current PCEnhancer code with an older version to get an idea how it can be done and to see some patterns I've used.
>>>>
>>>> As a quick hint: Java is basically a stack based virtual CPU. The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that comes the parameters. The size of the param on the stack depends on the type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
>>>> In case of a static method there is ofc no 'this' on the stack, so the parameters start with zero offset.
>>>> After all the parameters there are the unnamed 'local' variables. I tried to consistently name them xxxVarPos.
>>>> See also the various helper methods in ASMHelper. For example for the various LOAD, RETURN, STORE operations which depends based on the type (ALOAD vs ILOAD vs IALOAD etc)
>>>> Also read the guide to ASM PDF from objectweb.
>>>>
>>>> I'll be mostly offline till Sunday, but will try to answer mails if there are some questions.
>>>>
>>>> txs and LieGrue,
>>>> strub
>>>>
>>>>
>>>>
>>>>
>>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <ma...@gmail.com>:
>>>>>
>>>>> +1 thanks for tackling this, Mark!
>>>>>
>>>>> ASM definitely more widely used going forward.
>>>>>
>>>>> -Matt
>>>>>
>>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg <st...@yahoo.de.INVALID> wrote:
>>>>>>
>>>>>> Small update:
>>>>>>
>>>>>> I've worked on it over the last few weeks, and I'm getting closer
>>>>>>
>>>>>>
>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>
>>>>>> contains the latest work on the PCEnhancer. Right in the middle of replacing Serp with native ASM code.
>>>>>>
>>>>>> Note that I also had to modify a few interfaces with a few more to follow from BCClass to ASM ClassNode.
>>>>>>
>>>>>> Happy to get some feedback!
>>>>>>
>>>>>> If anybody wants to help with this effort I'm happy to also push this feature branch to our ASF repo. It looks reasonably promising already.
>>>>>>
>>>>>> LieGrue,
>>>>>> strub
>>>>>>
>>>>>>
>>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> Right now I'm trying to get rid of serp step by step.
>>>>>>> The code is right now at my own github repo in the fb_asmEnhance branch:
>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The approach I took for now is to create a few methods in AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other way around. That way we should be able to replace functionality part by part but still keep all things afloat.
>>>>>>>
>>>>>>> For now I've started with the PCSubclassValidator.
>>>>>>> Right now this evaluates the attributs using Serp plus ASM and then compare the results.
>>>>>>> If something is fishy, you'll see the following
>>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + " " + field);
>>>>>>>
>>>>>>>
>>>>>>> It would be extremely helpful, if you could go through your projects and let it run and report how it's going.
>>>>>>> If you see that "MSX ASMTODO" somewhere then we know I messed something up.
>>>>>>>
>>>>>>> I'll gonna finally remove the BCClass handling from those parts in a few days. Current commit is
>>>>>>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>>>>>>>
>>>>>>> txs and LieGrue,
>>>>>>> strub
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Re: enhance without Serp - Update
Posted by Mark Struberg <st...@yahoo.de.INVALID>.
Hi Francesco!
That's very valuable information!
Could you please do a git-bisect to find out which change did break it?
Actually the algorithm to select which classes are to be enhanced should not have changed. Thus I'm really curious what I did break.
Most times the amount of enhanced classes are the same but slightly differ in the generated code. That might be due to now we really generate Java11++ code whereas before we only generated Java 1.1 code (yes, not even 1.5!)
What I usually do is to have 2 checkouts. You might want to change the pom GAV version of the working checkout to something something different so you might have both in your ~/.m2/repository in parallel.
The first thing I do is to compare the class files in target/classes of both versions with 'compare with clipboard'.
I just open the class file in target/classes and let Intellij/Netbeans do the decompilation for me.
You might find some differences here in which case this needs further investigation. Note that the order of the methods might have changed.
Sometimes you will also see some 'cannot decompile' in the code block. This happens when something in the bytecode is messed up. You can also watch out for ClassNotFound or VerifyError, etc.
If this is the case I then use
$> javap -c -p -constants target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile
on both versions and check whether something is different.
Often it's a wrong offset in the stack calculation (e.g. ALOAD 1 vs ALOAD 2)
Note that the 'old' Java 1.1 bytecode doesn't understand LDC of Class constants (LoaD Constant). This was only introduced in Java 1.4 or 1.5 bytecode afair. Before that a dynamic Class.forName was called and stored in a static variable. Ugly stuff, and happy to finally get rid of all those hacks...
Hope that helps to get you started. Feel free to ping me again over here or at the #apachee ASF slack channel.
txs and LieGrue,
strub
> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò <il...@apache.org>:
>
> From a very preliminary analysis it seems that just a subset of the persistence-capable classes are effectively found and enhanced: now [2], it used to be [3].
>
> I'll continue investigating.
>
> Regards.
>
> On 03/07/23 13:56, Francesco Chicchiriccò wrote:
>> Hi Mark,
>> the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT is now failing - it used to work at least until last June 30th.
>>
>> The issues seem to be related to entity enhancement, which we perform via openjpa-maven-plugin - see [1].
>>
>> Is there anything obvious we should be changing?
>> Meanwhile I'll try to dig to see if I am able to understand something more.
>>
>> Regards.
>>
>> [1] https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174
>
> [2] https://paste.apache.org/gxcm6
> [3] https://paste.apache.org/1g9k5
>
>> On 03/07/23 12:15, Mark Struberg wrote:
>>> Update: I've now merged the wip to our master and pushed it to the ASF gitox repo.
>>>
>>> I think we already came pretty far and it looks really promising. So there is a high chance that we succeed, although there is still plenty of work in front of us.
>>>
>>> How it works:
>>>
>>> Basically there are 2 sets of information in the PCEnhancer right now:
>>>
>>> _managed: the BCClass version of the original class
>>> _pc: the BCClass of the generated/modified class. Might be the same as _managed but in case of subclassing or interface case it is different. In those cases _managed represents the original class and _pc the subclass/concrete implementation.
>>>
>>> managed and pc are the ASM variants of _managed and _pc.
>>>
>>> The main entry point for debugging into it is PCEnhancer#run()
>>>
>>> There are a few methods in ASMHelper to update from BCClass -> ASM and vice versa:
>>>
>>> AsmHelper.readIntoBCClass(pc, _pc);
>>> takes all the information from the ASM pc, creates a class byte[] and reads that back into the existing _pc instance.
>>>
>>> pc = AsmHelper.toClassNode(_pc);
>>> is the corresponding method in the other direction. from BCClass to ASM.
>>>
>>> Right now we have the following methods left to migrate in PCEnhancer
>>> addAttachDetachCode();
>>> addSerializationCode();
>>> addCloningCode();
>>> runAuxiliaryEnhancers();
>>>
>>>
>>> I'll be off for a week on holidays in Venice.
>>> If someone want's to join in with the effort then please go on! You might want to compare the current PCEnhancer code with an older version to get an idea how it can be done and to see some patterns I've used.
>>>
>>> As a quick hint: Java is basically a stack based virtual CPU. The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that comes the parameters. The size of the param on the stack depends on the type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
>>> In case of a static method there is ofc no 'this' on the stack, so the parameters start with zero offset.
>>> After all the parameters there are the unnamed 'local' variables. I tried to consistently name them xxxVarPos.
>>> See also the various helper methods in ASMHelper. For example for the various LOAD, RETURN, STORE operations which depends based on the type (ALOAD vs ILOAD vs IALOAD etc)
>>> Also read the guide to ASM PDF from objectweb.
>>>
>>> I'll be mostly offline till Sunday, but will try to answer mails if there are some questions.
>>>
>>> txs and LieGrue,
>>> strub
>>>
>>>
>>>
>>>
>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <ma...@gmail.com>:
>>>>
>>>> +1 thanks for tackling this, Mark!
>>>>
>>>> ASM definitely more widely used going forward.
>>>>
>>>> -Matt
>>>>
>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg <st...@yahoo.de.INVALID> wrote:
>>>>>
>>>>> Small update:
>>>>>
>>>>> I've worked on it over the last few weeks, and I'm getting closer
>>>>>
>>>>>
>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>
>>>>> contains the latest work on the PCEnhancer. Right in the middle of replacing Serp with native ASM code.
>>>>>
>>>>> Note that I also had to modify a few interfaces with a few more to follow from BCClass to ASM ClassNode.
>>>>>
>>>>> Happy to get some feedback!
>>>>>
>>>>> If anybody wants to help with this effort I'm happy to also push this feature branch to our ASF repo. It looks reasonably promising already.
>>>>>
>>>>> LieGrue,
>>>>> strub
>>>>>
>>>>>
>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> Right now I'm trying to get rid of serp step by step.
>>>>>> The code is right now at my own github repo in the fb_asmEnhance branch:
>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>>
>>>>>>
>>>>>>
>>>>>> The approach I took for now is to create a few methods in AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other way around. That way we should be able to replace functionality part by part but still keep all things afloat.
>>>>>>
>>>>>> For now I've started with the PCSubclassValidator.
>>>>>> Right now this evaluates the attributs using Serp plus ASM and then compare the results.
>>>>>> If something is fishy, you'll see the following
>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + " " + field);
>>>>>>
>>>>>>
>>>>>> It would be extremely helpful, if you could go through your projects and let it run and report how it's going.
>>>>>> If you see that "MSX ASMTODO" somewhere then we know I messed something up.
>>>>>>
>>>>>> I'll gonna finally remove the BCClass handling from those parts in a few days. Current commit is
>>>>>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>>>>>>
>>>>>> txs and LieGrue,
>>>>>> strub
>
> --
> Francesco Chicchiriccò
>
> Tirasa - Open Source Excellence
> http://www.tirasa.net/
>
> Member at The Apache Software Foundation
> Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
> http://home.apache.org/~ilgrosso/
Re: enhance without Serp - Update
Posted by Francesco Chicchiriccò <il...@apache.org>.
From a very preliminary analysis it seems that just a subset of the persistence-capable classes are effectively found and enhanced: now [2], it used to be [3].
I'll continue investigating.
Regards.
On 03/07/23 13:56, Francesco Chicchiriccò wrote:
> Hi Mark,
> the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT is now failing - it used to work at least until last June 30th.
>
> The issues seem to be related to entity enhancement, which we perform via openjpa-maven-plugin - see [1].
>
> Is there anything obvious we should be changing?
> Meanwhile I'll try to dig to see if I am able to understand something more.
>
> Regards.
>
> [1] https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174
[2] https://paste.apache.org/gxcm6
[3] https://paste.apache.org/1g9k5
> On 03/07/23 12:15, Mark Struberg wrote:
>> Update: I've now merged the wip to our master and pushed it to the ASF gitox repo.
>>
>> I think we already came pretty far and it looks really promising. So there is a high chance that we succeed, although there is still plenty of work in front of us.
>>
>> How it works:
>>
>> Basically there are 2 sets of information in the PCEnhancer right now:
>>
>> _managed: the BCClass version of the original class
>> _pc: the BCClass of the generated/modified class. Might be the same as _managed but in case of subclassing or interface case it is different. In those cases _managed represents the original class and _pc the subclass/concrete implementation.
>>
>> managed and pc are the ASM variants of _managed and _pc.
>>
>> The main entry point for debugging into it is PCEnhancer#run()
>>
>> There are a few methods in ASMHelper to update from BCClass -> ASM and vice versa:
>>
>> AsmHelper.readIntoBCClass(pc, _pc);
>> takes all the information from the ASM pc, creates a class byte[] and reads that back into the existing _pc instance.
>>
>> pc = AsmHelper.toClassNode(_pc);
>> is the corresponding method in the other direction. from BCClass to ASM.
>>
>> Right now we have the following methods left to migrate in PCEnhancer
>> addAttachDetachCode();
>> addSerializationCode();
>> addCloningCode();
>> runAuxiliaryEnhancers();
>>
>>
>> I'll be off for a week on holidays in Venice.
>> If someone want's to join in with the effort then please go on! You might want to compare the current PCEnhancer code with an older version to get an idea how it can be done and to see some patterns I've used.
>>
>> As a quick hint: Java is basically a stack based virtual CPU. The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that comes the parameters. The size of the param on the stack depends on the type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
>> In case of a static method there is ofc no 'this' on the stack, so the parameters start with zero offset.
>> After all the parameters there are the unnamed 'local' variables. I tried to consistently name them xxxVarPos.
>> See also the various helper methods in ASMHelper. For example for the various LOAD, RETURN, STORE operations which depends based on the type (ALOAD vs ILOAD vs IALOAD etc)
>> Also read the guide to ASM PDF from objectweb.
>>
>> I'll be mostly offline till Sunday, but will try to answer mails if there are some questions.
>>
>> txs and LieGrue,
>> strub
>>
>>
>>
>>
>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <ma...@gmail.com>:
>>>
>>> +1 thanks for tackling this, Mark!
>>>
>>> ASM definitely more widely used going forward.
>>>
>>> -Matt
>>>
>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg <st...@yahoo.de.INVALID> wrote:
>>>>
>>>> Small update:
>>>>
>>>> I've worked on it over the last few weeks, and I'm getting closer
>>>>
>>>>
>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>
>>>> contains the latest work on the PCEnhancer. Right in the middle of replacing Serp with native ASM code.
>>>>
>>>> Note that I also had to modify a few interfaces with a few more to follow from BCClass to ASM ClassNode.
>>>>
>>>> Happy to get some feedback!
>>>>
>>>> If anybody wants to help with this effort I'm happy to also push this feature branch to our ASF repo. It looks reasonably promising already.
>>>>
>>>> LieGrue,
>>>> strub
>>>>
>>>>
>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>>>>
>>>>> Hi!
>>>>>
>>>>> Right now I'm trying to get rid of serp step by step.
>>>>> The code is right now at my own github repo in the fb_asmEnhance branch:
>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>>
>>>>>
>>>>>
>>>>> The approach I took for now is to create a few methods in AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other way around. That way we should be able to replace functionality part by part but still keep all things afloat.
>>>>>
>>>>> For now I've started with the PCSubclassValidator.
>>>>> Right now this evaluates the attributs using Serp plus ASM and then compare the results.
>>>>> If something is fishy, you'll see the following
>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + " " + field);
>>>>>
>>>>>
>>>>> It would be extremely helpful, if you could go through your projects and let it run and report how it's going.
>>>>> If you see that "MSX ASMTODO" somewhere then we know I messed something up.
>>>>>
>>>>> I'll gonna finally remove the BCClass handling from those parts in a few days. Current commit is
>>>>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>>>>>
>>>>> txs and LieGrue,
>>>>> strub
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Re: enhance without Serp - Update
Posted by Francesco Chicchiriccò <il...@apache.org>.
Hi Mark,
the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT is now failing - it used to work at least until last June 30th.
The issues seem to be related to entity enhancement, which we perform via openjpa-maven-plugin - see [1].
Is there anything obvious we should be changing?
Meanwhile I'll try to dig to see if I am able to understand something more.
Regards.
[1] https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174
On 03/07/23 12:15, Mark Struberg wrote:
> Update: I've now merged the wip to our master and pushed it to the ASF gitox repo.
>
> I think we already came pretty far and it looks really promising. So there is a high chance that we succeed, although there is still plenty of work in front of us.
>
> How it works:
>
> Basically there are 2 sets of information in the PCEnhancer right now:
>
> _managed: the BCClass version of the original class
> _pc: the BCClass of the generated/modified class. Might be the same as _managed but in case of subclassing or interface case it is different. In those cases _managed represents the original class and _pc the subclass/concrete implementation.
>
> managed and pc are the ASM variants of _managed and _pc.
>
> The main entry point for debugging into it is PCEnhancer#run()
>
> There are a few methods in ASMHelper to update from BCClass -> ASM and vice versa:
>
> AsmHelper.readIntoBCClass(pc, _pc);
> takes all the information from the ASM pc, creates a class byte[] and reads that back into the existing _pc instance.
>
> pc = AsmHelper.toClassNode(_pc);
> is the corresponding method in the other direction. from BCClass to ASM.
>
> Right now we have the following methods left to migrate in PCEnhancer
> addAttachDetachCode();
> addSerializationCode();
> addCloningCode();
> runAuxiliaryEnhancers();
>
>
> I'll be off for a week on holidays in Venice.
> If someone want's to join in with the effort then please go on! You might want to compare the current PCEnhancer code with an older version to get an idea how it can be done and to see some patterns I've used.
>
> As a quick hint: Java is basically a stack based virtual CPU. The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that comes the parameters. The size of the param on the stack depends on the type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
> In case of a static method there is ofc no 'this' on the stack, so the parameters start with zero offset.
> After all the parameters there are the unnamed 'local' variables. I tried to consistently name them xxxVarPos.
> See also the various helper methods in ASMHelper. For example for the various LOAD, RETURN, STORE operations which depends based on the type (ALOAD vs ILOAD vs IALOAD etc)
> Also read the guide to ASM PDF from objectweb.
>
> I'll be mostly offline till Sunday, but will try to answer mails if there are some questions.
>
> txs and LieGrue,
> strub
>
>
>
>
>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <ma...@gmail.com>:
>>
>> +1 thanks for tackling this, Mark!
>>
>> ASM definitely more widely used going forward.
>>
>> -Matt
>>
>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg <st...@yahoo.de.INVALID> wrote:
>>>
>>> Small update:
>>>
>>> I've worked on it over the last few weeks, and I'm getting closer
>>>
>>>
>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>
>>> contains the latest work on the PCEnhancer. Right in the middle of replacing Serp with native ASM code.
>>>
>>> Note that I also had to modify a few interfaces with a few more to follow from BCClass to ASM ClassNode.
>>>
>>> Happy to get some feedback!
>>>
>>> If anybody wants to help with this effort I'm happy to also push this feature branch to our ASF repo. It looks reasonably promising already.
>>>
>>> LieGrue,
>>> strub
>>>
>>>
>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>>>
>>>> Hi!
>>>>
>>>> Right now I'm trying to get rid of serp step by step.
>>>> The code is right now at my own github repo in the fb_asmEnhance branch:
>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>>
>>>>
>>>>
>>>> The approach I took for now is to create a few methods in AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other way around. That way we should be able to replace functionality part by part but still keep all things afloat.
>>>>
>>>> For now I've started with the PCSubclassValidator.
>>>> Right now this evaluates the attributs using Serp plus ASM and then compare the results.
>>>> If something is fishy, you'll see the following
>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + " " + field);
>>>>
>>>>
>>>> It would be extremely helpful, if you could go through your projects and let it run and report how it's going.
>>>> If you see that "MSX ASMTODO" somewhere then we know I messed something up.
>>>>
>>>> I'll gonna finally remove the BCClass handling from those parts in a few days. Current commit is
>>>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>>>>
>>>> txs and LieGrue,
>>>> strub
>>>>
>>>>
>>>>
>>>>
>
--
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Re: enhance without Serp - Update
Posted by Mark Struberg <st...@yahoo.de.INVALID>.
Update: I've now merged the wip to our master and pushed it to the ASF gitox repo.
I think we already came pretty far and it looks really promising. So there is a high chance that we succeed, although there is still plenty of work in front of us.
How it works:
Basically there are 2 sets of information in the PCEnhancer right now:
_managed: the BCClass version of the original class
_pc: the BCClass of the generated/modified class. Might be the same as _managed but in case of subclassing or interface case it is different. In those cases _managed represents the original class and _pc the subclass/concrete implementation.
managed and pc are the ASM variants of _managed and _pc.
The main entry point for debugging into it is PCEnhancer#run()
There are a few methods in ASMHelper to update from BCClass -> ASM and vice versa:
AsmHelper.readIntoBCClass(pc, _pc);
takes all the information from the ASM pc, creates a class byte[] and reads that back into the existing _pc instance.
pc = AsmHelper.toClassNode(_pc);
is the corresponding method in the other direction. from BCClass to ASM.
Right now we have the following methods left to migrate in PCEnhancer
addAttachDetachCode();
addSerializationCode();
addCloningCode();
runAuxiliaryEnhancers();
I'll be off for a week on holidays in Venice.
If someone want's to join in with the effort then please go on! You might want to compare the current PCEnhancer code with an older version to get an idea how it can be done and to see some patterns I've used.
As a quick hint: Java is basically a stack based virtual CPU. The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that comes the parameters. The size of the param on the stack depends on the type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type).
In case of a static method there is ofc no 'this' on the stack, so the parameters start with zero offset.
After all the parameters there are the unnamed 'local' variables. I tried to consistently name them xxxVarPos.
See also the various helper methods in ASMHelper. For example for the various LOAD, RETURN, STORE operations which depends based on the type (ALOAD vs ILOAD vs IALOAD etc)
Also read the guide to ASM PDF from objectweb.
I'll be mostly offline till Sunday, but will try to answer mails if there are some questions.
txs and LieGrue,
strub
> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich <ma...@gmail.com>:
>
> +1 thanks for tackling this, Mark!
>
> ASM definitely more widely used going forward.
>
> -Matt
>
>> On Jun 17, 2023, at 3:01 PM, Mark Struberg <st...@yahoo.de.INVALID> wrote:
>>
>> Small update:
>>
>> I've worked on it over the last few weeks, and I'm getting closer
>>
>>
>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>
>> contains the latest work on the PCEnhancer. Right in the middle of replacing Serp with native ASM code.
>>
>> Note that I also had to modify a few interfaces with a few more to follow from BCClass to ASM ClassNode.
>>
>> Happy to get some feedback!
>>
>> If anybody wants to help with this effort I'm happy to also push this feature branch to our ASF repo. It looks reasonably promising already.
>>
>> LieGrue,
>> strub
>>
>>
>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>>
>>> Hi!
>>>
>>> Right now I'm trying to get rid of serp step by step.
>>> The code is right now at my own github repo in the fb_asmEnhance branch:
>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>>
>>>
>>>
>>> The approach I took for now is to create a few methods in AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other way around. That way we should be able to replace functionality part by part but still keep all things afloat.
>>>
>>> For now I've started with the PCSubclassValidator.
>>> Right now this evaluates the attributs using Serp plus ASM and then compare the results.
>>> If something is fishy, you'll see the following
>>> throw new IllegalStateException("MSX ASMTODO " + bcField + " " + field);
>>>
>>>
>>> It would be extremely helpful, if you could go through your projects and let it run and report how it's going.
>>> If you see that "MSX ASMTODO" somewhere then we know I messed something up.
>>>
>>> I'll gonna finally remove the BCClass handling from those parts in a few days. Current commit is
>>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>>>
>>> txs and LieGrue,
>>> strub
>>>
>>>
>>>
>>>
>>
>
Re: enhance without Serp
Posted by Matt Pavlovich <ma...@gmail.com>.
+1 thanks for tackling this, Mark!
ASM definitely more widely used going forward.
-Matt
> On Jun 17, 2023, at 3:01 PM, Mark Struberg <st...@yahoo.de.INVALID> wrote:
>
> Small update:
>
> I've worked on it over the last few weeks, and I'm getting closer
>
>
> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>
> contains the latest work on the PCEnhancer. Right in the middle of replacing Serp with native ASM code.
>
> Note that I also had to modify a few interfaces with a few more to follow from BCClass to ASM ClassNode.
>
> Happy to get some feedback!
>
> If anybody wants to help with this effort I'm happy to also push this feature branch to our ASF repo. It looks reasonably promising already.
>
> LieGrue,
> strub
>
>
>> Am 25.05.2023 um 18:36 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>>
>> Hi!
>>
>> Right now I'm trying to get rid of serp step by step.
>> The code is right now at my own github repo in the fb_asmEnhance branch:
>> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>>
>>
>>
>> The approach I took for now is to create a few methods in AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other way around. That way we should be able to replace functionality part by part but still keep all things afloat.
>>
>> For now I've started with the PCSubclassValidator.
>> Right now this evaluates the attributs using Serp plus ASM and then compare the results.
>> If something is fishy, you'll see the following
>> throw new IllegalStateException("MSX ASMTODO " + bcField + " " + field);
>>
>>
>> It would be extremely helpful, if you could go through your projects and let it run and report how it's going.
>> If you see that "MSX ASMTODO" somewhere then we know I messed something up.
>>
>> I'll gonna finally remove the BCClass handling from those parts in a few days. Current commit is
>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>>
>> txs and LieGrue,
>> strub
>>
>>
>>
>>
>
Re: enhance without Serp
Posted by Mark Struberg <st...@yahoo.de.INVALID>.
Small update:
I've worked on it over the last few weeks, and I'm getting closer
https://github.com/struberg/openjpa/tree/fb_asmEnhance
contains the latest work on the PCEnhancer. Right in the middle of replacing Serp with native ASM code.
Note that I also had to modify a few interfaces with a few more to follow from BCClass to ASM ClassNode.
Happy to get some feedback!
If anybody wants to help with this effort I'm happy to also push this feature branch to our ASF repo. It looks reasonably promising already.
LieGrue,
strub
> Am 25.05.2023 um 18:36 schrieb Mark Struberg <st...@yahoo.de.INVALID>:
>
> Hi!
>
> Right now I'm trying to get rid of serp step by step.
> The code is right now at my own github repo in the fb_asmEnhance branch:
> https://github.com/struberg/openjpa/tree/fb_asmEnhance
>
>
>
> The approach I took for now is to create a few methods in AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other way around. That way we should be able to replace functionality part by part but still keep all things afloat.
>
> For now I've started with the PCSubclassValidator.
> Right now this evaluates the attributs using Serp plus ASM and then compare the results.
> If something is fishy, you'll see the following
> throw new IllegalStateException("MSX ASMTODO " + bcField + " " + field);
>
>
> It would be extremely helpful, if you could go through your projects and let it run and report how it's going.
> If you see that "MSX ASMTODO" somewhere then we know I messed something up.
>
> I'll gonna finally remove the BCClass handling from those parts in a few days. Current commit is
> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b
>
> txs and LieGrue,
> strub
>
>
>
>