You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@groovy.apache.org by Saravanan Palanichamy <ch...@gmail.com> on 2020/08/08 13:19:01 UTC
Upgrading from 2.5 to 3.0.5 and my class loader extension does not work
Hello
I have an extension of GroovyClassLoader that I use as follows
a) I override loadClass to make sure I decide where to load from
b) I first call loadClass on my parent class loader to make sure the class is not loaded there (I assert if it is. This code is not shown below)
c) I then call loadClass after turning off all script loading on myclassloader to make sure this class is not already loaded. If it is, I just use that
d) If I get a class not found exception, I know this class is not already loaded and so I look up a local repository to fetch a string based script and call parseClass (I do all of this in the context of loadClass)
This used to work with 2.5.x but when I upgrade to 3.0.5, it sits in a blocked state (not sure if it is blocked on some mutex because of a recursion some where). When I use my debugger , when it gets to this state, I am not able to read any of my variable values in the debugger (which seems to indicate some threading read lock somewhere)
Question :- Am I doing this the right way? Why does it block and not load the class? It locks up in the parseClass invocation (inside a validate call that validates the sourceCode object
My code looks like this
class MyClassLoader constructor(
private val repository: FlowsRepository,
private val configuration: CompilerConfiguration,
parentClassLoader: ClassLoader
) : GroovyClassLoader(parentClassLoader, configuration) {
/** Load class [name] either from repository or class path */
override fun loadClass(
name: String,
lookupScriptFiles: Boolean,
preferClassOverScript: Boolean,
resolve: Boolean
): Class<*> {
return try {
// Never try to load from script files, always parse from repository
super.loadClass(name, false, true, resolve)
} catch (e: ClassNotFoundException) {
// Try fetching it from the repository
parseFromRepository(name)
}
}
/** Parse [name] from the flow repository and cache it */
private fun parseFromRepository(name: String): Class<*> {
// Fetch flow source
val flow = repository.fetchFlow(name) ?: throw ClassNotFoundException("Unable to find flow $name")
// Parse the class if found
try {
// If parseClass yields a class whose name does not match (typically when we are trying to find an inner
// class (A.B) by parsing the outer class source file (A.groovy), throw class not found. The compiler will
// attempt to then find A$B which is how inner classes are stored in the class loader. This find will
// succeed (this process is repeated for inner classes inside inner classes)
super.parseClass(flow.sourceCode, true)
// The class parsed may not be the class you are looking for, but it may be an inner class, so
// use the cache to get it
return getClassCacheEntry(name) ?: throw ClassNotFoundException("Unable to find flow $name")
} catch (e: CompilationFailedException) {
throw CodeQualityException("Compile error", e)
}
}
}
Re: Upgrading from 2.5 to 3.0.5 and my class loader extension does not work
Posted by Saravanan Palanichamy <ch...@gmail.com>.
Apologies to the group
This was a bad dependency merge of junit that was causing this thread lockup. It works just fine now
regards
Saravanan
On 2020/08/10 04:41:42, Daniel Sun <su...@apache.org> wrote:
> Could you provide a standalone runnable script to reproduce the issue?
> BTW, the script is expected to be written in Groovy/Java.
>
> Cheers,
> Daniel Sun
> On 2020/08/08 13:19:01, Saravanan Palanichamy <ch...@gmail.com> wrote:
> > Hello
> >
> > I have an extension of GroovyClassLoader that I use as follows
> >
> > a) I override loadClass to make sure I decide where to load from
> > b) I first call loadClass on my parent class loader to make sure the class is not loaded there (I assert if it is. This code is not shown below)
> > c) I then call loadClass after turning off all script loading on myclassloader to make sure this class is not already loaded. If it is, I just use that
> > d) If I get a class not found exception, I know this class is not already loaded and so I look up a local repository to fetch a string based script and call parseClass (I do all of this in the context of loadClass)
> >
> > This used to work with 2.5.x but when I upgrade to 3.0.5, it sits in a blocked state (not sure if it is blocked on some mutex because of a recursion some where). When I use my debugger , when it gets to this state, I am not able to read any of my variable values in the debugger (which seems to indicate some threading read lock somewhere)
> >
> > Question :- Am I doing this the right way? Why does it block and not load the class? It locks up in the parseClass invocation (inside a validate call that validates the sourceCode object
> >
> > My code looks like this
> >
> > class MyClassLoader constructor(
> > private val repository: FlowsRepository,
> > private val configuration: CompilerConfiguration,
> > parentClassLoader: ClassLoader
> > ) : GroovyClassLoader(parentClassLoader, configuration) {
> >
> > /** Load class [name] either from repository or class path */
> > override fun loadClass(
> > name: String,
> > lookupScriptFiles: Boolean,
> > preferClassOverScript: Boolean,
> > resolve: Boolean
> > ): Class<*> {
> > return try {
> > // Never try to load from script files, always parse from repository
> > super.loadClass(name, false, true, resolve)
> > } catch (e: ClassNotFoundException) {
> > // Try fetching it from the repository
> > parseFromRepository(name)
> > }
> > }
> >
> > /** Parse [name] from the flow repository and cache it */
> > private fun parseFromRepository(name: String): Class<*> {
> >
> > // Fetch flow source
> > val flow = repository.fetchFlow(name) ?: throw ClassNotFoundException("Unable to find flow $name")
> >
> > // Parse the class if found
> > try {
> > // If parseClass yields a class whose name does not match (typically when we are trying to find an inner
> > // class (A.B) by parsing the outer class source file (A.groovy), throw class not found. The compiler will
> > // attempt to then find A$B which is how inner classes are stored in the class loader. This find will
> > // succeed (this process is repeated for inner classes inside inner classes)
> > super.parseClass(flow.sourceCode, true)
> >
> > // The class parsed may not be the class you are looking for, but it may be an inner class, so
> > // use the cache to get it
> > return getClassCacheEntry(name) ?: throw ClassNotFoundException("Unable to find flow $name")
> > } catch (e: CompilationFailedException) {
> > throw CodeQualityException("Compile error", e)
> > }
> > }
> > }
> >
>
Re: Upgrading from 2.5 to 3.0.5 and my class loader extension does not work
Posted by Daniel Sun <su...@apache.org>.
Hi Saravanan,
Sorry for the late reply... I am too busy recently...
Yep, `StampedCommonCache` is not reentrant. You could try to replace `StampedCommonCache ` with `org.codehaus.groovy.runtime.memoize.UnlimitedConcurrentCache`, and build Groovy via `gradlew clean installGroovy --no-build-cache` to get the binary distribution.
If it works, PR is welcomed. Good luck ;-)
Cheers,
Daniel Sun
On 2020/08/17 10:02:12, Saravanan Palanichamy <ch...@gmail.com> wrote:
> Digging into this further, It looks like StampedCommonCache is not
> reentrant. But in my case, I am calling loadClass from inside another
> loadClass. Is this the problem? What is the recommendation to compile a set
> of classes together that have dependencies between each other?
>
> In the attachment, notice how loadClass is called within loadClass. This is
> because one of my import statements in the first class requires a second
> class load
>
> regards
> Saravanan
>
> On Mon, Aug 17, 2020 at 11:33 AM Saravanan Palanichamy <ch...@gmail.com>
> wrote:
>
> > While debugging this, I noticed this. Intellij tells me that the source
> > file and class file dont match. The debugging also has a weird jump from
> > GroovyClassLoader::doParseClass() line number 366 to
> > StampedCommonCache::isEmpty. I am using JAVA 11 and Groovy 3.0.5
> >
>
Re: Upgrading from 2.5 to 3.0.5 and my class loader extension does
not work
Posted by Saravanan Palanichamy <ch...@gmail.com>.
Digging into this further, It looks like StampedCommonCache is not
reentrant. But in my case, I am calling loadClass from inside another
loadClass. Is this the problem? What is the recommendation to compile a set
of classes together that have dependencies between each other?
In the attachment, notice how loadClass is called within loadClass. This is
because one of my import statements in the first class requires a second
class load
regards
Saravanan
On Mon, Aug 17, 2020 at 11:33 AM Saravanan Palanichamy <ch...@gmail.com>
wrote:
> While debugging this, I noticed this. Intellij tells me that the source
> file and class file dont match. The debugging also has a weird jump from
> GroovyClassLoader::doParseClass() line number 366 to
> StampedCommonCache::isEmpty. I am using JAVA 11 and Groovy 3.0.5
>
Re: Upgrading from 2.5 to 3.0.5 and my class loader extension does
not work
Posted by Saravanan Palanichamy <ch...@gmail.com>.
While debugging this, I noticed this. Intellij tells me that the source
file and class file dont match. The debugging also has a weird jump from
GroovyClassLoader::doParseClass() line number 366 to
StampedCommonCache::isEmpty. I am using JAVA 11 and Groovy 3.0.5
Re: Upgrading from 2.5 to 3.0.5 and my class loader extension does
not work
Posted by Saravanan Palanichamy <ch...@gmail.com>.
Hello Daniel
I was able to reproduce this error. I am still not sure if it is my setup
that is causing this error. I have attached my code files as a zip file
My class loader code is this. Notice that I overload the loadClass method
and if I hit a class not found exception, I try to parse the class through
a second parseClass method.
The problem I see is this. I am importing another class (Pojo) from the
first groovy file. So when I compile the first file, internally it tries to
resolve Pojo and tries to compile the second file. Which is where it hits
the getAndPut call in sourceCache.
This works in 2.5 because it is using a concurrentcommoncache for
sourceCache, whereas 3.0.5 is using a StampedCommonCache which does not
allow recursion on getAndPut. The logs I see in my console show I am
attempting to load the top most class and its import class one after the
other. My unit test is also added towards the end of this file
public class MyClassLoader extends GroovyClassLoader {
private FlowRepository repository;
private CompilerConfiguration configuration;
public MyClassLoader(ClassLoader parentClassLoader,
CompilerConfiguration configuration, FlowRepository repository) {
super(parentClassLoader, configuration);
this.repository = repository;
}
@Override
public Class loadClass(
String name,
boolean lookupScriptFiles,
boolean preferClassOverScript,
boolean resolve
) throws ClassNotFoundException, CompilationFailedException {
try {
// Never try to load from script files, always parse from repository
return super.loadClass(name, false, true, resolve);
} catch (ClassNotFoundException e) {
// Try fetching it from the repository
return parseFromRepository(name);
}
}
/**
* Parse [name] from the flow repository and cache it
*/
private Class parseFromRepository(String name) throws
UnsupportedOperationException, ClassNotFoundException {
// Fetch flow source
Flow flow = repository.fetchFlow(name);
if (flow == null)
throw new ClassNotFoundException("Unable to find flow $name");
// Parse the class if found
try {
System.out.println("Loading class " + flow.sourceCode.getName());
// If parseClass yields a class whose name does not match
(typically when we are trying to find an inner
// class (A.B) by parsing the outer class source file
(A.groovy), throw class not found. The compiler will
// attempt to then find A$B which is how inner classes are
stored in the class loader. This find will
// succeed (this process is repeated for inner classes
inside inner classes)
super.parseClass(flow.sourceCode, true);
System.out.println("Finished loading class " +
flow.sourceCode.getName());
// The class parsed may not be the class you are looking
for, but it may be an inner class, so
// use the cache to get it
Class cachedClass = super.getClassCacheEntry(name);
if (cachedClass != null)
return cachedClass;
else
throw new ClassNotFoundException("Unable to find flow $name");
} catch (CompilationFailedException e) {
throw new UnsupportedOperationException("Compile error", e);
}
}
}
@Test
fun testMe() {
val repository = FlowFolderRepository()
val config = CompilerConfiguration().addCompilationCustomizers(
ASTTransformationCustomizer(CompileStatic::class.java)
)
val sandbox = MyClassLoader(this::class.java.classLoader, config,
repository)
// Load all flows in the repository
for (flow in repository.fetchAllFlows()) {
// Make sure class is not already loaded in the parent class
loader. This is usually a Config file error
try {
this::class.java.classLoader.loadClass(flow.name)
throw ConfigQualityException("${flow.name} is already
loaded and cannot be recompiled.")
} catch (e: ClassNotFoundException) {
}
// Load the class
sandbox.loadClass(flow.name)
}
}
On Mon, Aug 10, 2020 at 10:11 AM Daniel Sun <su...@apache.org> wrote:
> Could you provide a standalone runnable script to reproduce the issue?
> BTW, the script is expected to be written in Groovy/Java.
>
> Cheers,
> Daniel Sun
> On 2020/08/08 13:19:01, Saravanan Palanichamy <ch...@gmail.com> wrote:
> > Hello
> >
> > I have an extension of GroovyClassLoader that I use as follows
> >
> > a) I override loadClass to make sure I decide where to load from
> > b) I first call loadClass on my parent class loader to make sure the
> class is not loaded there (I assert if it is. This code is not shown below)
> > c) I then call loadClass after turning off all script loading on
> myclassloader to make sure this class is not already loaded. If it is, I
> just use that
> > d) If I get a class not found exception, I know this class is not
> already loaded and so I look up a local repository to fetch a string based
> script and call parseClass (I do all of this in the context of loadClass)
> >
> > This used to work with 2.5.x but when I upgrade to 3.0.5, it sits in a
> blocked state (not sure if it is blocked on some mutex because of a
> recursion some where). When I use my debugger , when it gets to this state,
> I am not able to read any of my variable values in the debugger (which
> seems to indicate some threading read lock somewhere)
> >
> > Question :- Am I doing this the right way? Why does it block and not
> load the class? It locks up in the parseClass invocation (inside a validate
> call that validates the sourceCode object
> >
> > My code looks like this
> >
> > class MyClassLoader constructor(
> > private val repository: FlowsRepository,
> > private val configuration: CompilerConfiguration,
> > parentClassLoader: ClassLoader
> > ) : GroovyClassLoader(parentClassLoader, configuration) {
> >
> > /** Load class [name] either from repository or class path */
> > override fun loadClass(
> > name: String,
> > lookupScriptFiles: Boolean,
> > preferClassOverScript: Boolean,
> > resolve: Boolean
> > ): Class<*> {
> > return try {
> > // Never try to load from script files, always parse from
> repository
> > super.loadClass(name, false, true, resolve)
> > } catch (e: ClassNotFoundException) {
> > // Try fetching it from the repository
> > parseFromRepository(name)
> > }
> > }
> >
> > /** Parse [name] from the flow repository and cache it */
> > private fun parseFromRepository(name: String): Class<*> {
> >
> > // Fetch flow source
> > val flow = repository.fetchFlow(name) ?: throw
> ClassNotFoundException("Unable to find flow $name")
> >
> > // Parse the class if found
> > try {
> > // If parseClass yields a class whose name does not match
> (typically when we are trying to find an inner
> > // class (A.B) by parsing the outer class source file
> (A.groovy), throw class not found. The compiler will
> > // attempt to then find A$B which is how inner classes are
> stored in the class loader. This find will
> > // succeed (this process is repeated for inner classes
> inside inner classes)
> > super.parseClass(flow.sourceCode, true)
> >
> > // The class parsed may not be the class you are looking
> for, but it may be an inner class, so
> > // use the cache to get it
> > return getClassCacheEntry(name) ?: throw
> ClassNotFoundException("Unable to find flow $name")
> > } catch (e: CompilationFailedException) {
> > throw CodeQualityException("Compile error", e)
> > }
> > }
> > }
> >
>
Re: Upgrading from 2.5 to 3.0.5 and my class loader extension does not work
Posted by Daniel Sun <su...@apache.org>.
Could you provide a standalone runnable script to reproduce the issue?
BTW, the script is expected to be written in Groovy/Java.
Cheers,
Daniel Sun
On 2020/08/08 13:19:01, Saravanan Palanichamy <ch...@gmail.com> wrote:
> Hello
>
> I have an extension of GroovyClassLoader that I use as follows
>
> a) I override loadClass to make sure I decide where to load from
> b) I first call loadClass on my parent class loader to make sure the class is not loaded there (I assert if it is. This code is not shown below)
> c) I then call loadClass after turning off all script loading on myclassloader to make sure this class is not already loaded. If it is, I just use that
> d) If I get a class not found exception, I know this class is not already loaded and so I look up a local repository to fetch a string based script and call parseClass (I do all of this in the context of loadClass)
>
> This used to work with 2.5.x but when I upgrade to 3.0.5, it sits in a blocked state (not sure if it is blocked on some mutex because of a recursion some where). When I use my debugger , when it gets to this state, I am not able to read any of my variable values in the debugger (which seems to indicate some threading read lock somewhere)
>
> Question :- Am I doing this the right way? Why does it block and not load the class? It locks up in the parseClass invocation (inside a validate call that validates the sourceCode object
>
> My code looks like this
>
> class MyClassLoader constructor(
> private val repository: FlowsRepository,
> private val configuration: CompilerConfiguration,
> parentClassLoader: ClassLoader
> ) : GroovyClassLoader(parentClassLoader, configuration) {
>
> /** Load class [name] either from repository or class path */
> override fun loadClass(
> name: String,
> lookupScriptFiles: Boolean,
> preferClassOverScript: Boolean,
> resolve: Boolean
> ): Class<*> {
> return try {
> // Never try to load from script files, always parse from repository
> super.loadClass(name, false, true, resolve)
> } catch (e: ClassNotFoundException) {
> // Try fetching it from the repository
> parseFromRepository(name)
> }
> }
>
> /** Parse [name] from the flow repository and cache it */
> private fun parseFromRepository(name: String): Class<*> {
>
> // Fetch flow source
> val flow = repository.fetchFlow(name) ?: throw ClassNotFoundException("Unable to find flow $name")
>
> // Parse the class if found
> try {
> // If parseClass yields a class whose name does not match (typically when we are trying to find an inner
> // class (A.B) by parsing the outer class source file (A.groovy), throw class not found. The compiler will
> // attempt to then find A$B which is how inner classes are stored in the class loader. This find will
> // succeed (this process is repeated for inner classes inside inner classes)
> super.parseClass(flow.sourceCode, true)
>
> // The class parsed may not be the class you are looking for, but it may be an inner class, so
> // use the cache to get it
> return getClassCacheEntry(name) ?: throw ClassNotFoundException("Unable to find flow $name")
> } catch (e: CompilationFailedException) {
> throw CodeQualityException("Compile error", e)
> }
> }
> }
>