Upgrading from 2.5 to 3.0.5 and my class loader extension does not work

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Upgrading from 2.5 to 3.0.5 and my class loader extension does not work

Saravanan Palanichamy
Hello

I have an extension of GroovyClassLoader that I use as follows

a) I override loadClass to make sure I decide where to load from
b) I first call loadClass on my parent class loader to make sure the class is not loaded there (I assert if it is. This code is not shown below)
c) I then call loadClass after turning off all script loading on myclassloader to make sure this class is not already loaded. If it is, I just use that
d) If I get a class not found exception, I know this class is not already loaded and so I look up a local repository to fetch a string based script and call parseClass (I do all of this in the context of loadClass)

This used to work with 2.5.x but when I upgrade to 3.0.5, it sits in a blocked state (not sure if it is blocked on some mutex because of a recursion some where). When I use my debugger , when it gets to this state, I am not able to read any of my variable values in the debugger (which seems to indicate some threading read lock somewhere)

Question :- Am I doing this the right way? Why does it block and not load the class? It locks up in the parseClass invocation (inside a validate call that validates the sourceCode object

My code looks like this

class MyClassLoader constructor(
    private val repository: FlowsRepository,
    private val configuration: CompilerConfiguration,
    parentClassLoader: ClassLoader
) : GroovyClassLoader(parentClassLoader, configuration) {

    /** Load class [name] either from repository or class path */
    override fun loadClass(
        name: String,
        lookupScriptFiles: Boolean,
        preferClassOverScript: Boolean,
        resolve: Boolean
    ): Class<*> {
        return try {
            // Never try to load from script files, always parse from repository
            super.loadClass(name, false, true, resolve)
        } catch (e: ClassNotFoundException) {
            // Try fetching it from the repository
            parseFromRepository(name)
        }
    }

    /** Parse [name] from the flow repository and cache it */
    private fun parseFromRepository(name: String): Class<*> {

        // Fetch flow source
        val flow = repository.fetchFlow(name) ?: throw ClassNotFoundException("Unable to find flow $name")

        // Parse the class if found
        try {
            // If parseClass yields a class whose name does not match (typically when we are trying to find an inner
            // class (A.B) by parsing the outer class source file (A.groovy), throw class not found. The compiler will
            // attempt to then find A$B which is how inner classes are stored in the class loader. This find will
            // succeed (this process is repeated for inner classes inside inner classes)
            super.parseClass(flow.sourceCode, true)

            // The class parsed may not be the class you are looking for, but it may be an inner class, so
            // use the cache to get it
            return getClassCacheEntry(name) ?: throw ClassNotFoundException("Unable to find flow $name")
        } catch (e: CompilationFailedException) {
            throw CodeQualityException("Compile error", e)
        }
    }
}
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading from 2.5 to 3.0.5 and my class loader extension does not work

Daniel.Sun
Could you provide a standalone runnable script to reproduce the issue?
BTW, the script is expected to be written in Groovy/Java.

Cheers,
Daniel Sun
On 2020/08/08 13:19:01, Saravanan Palanichamy <[hidden email]> wrote:

> Hello
>
> I have an extension of GroovyClassLoader that I use as follows
>
> a) I override loadClass to make sure I decide where to load from
> b) I first call loadClass on my parent class loader to make sure the class is not loaded there (I assert if it is. This code is not shown below)
> c) I then call loadClass after turning off all script loading on myclassloader to make sure this class is not already loaded. If it is, I just use that
> d) If I get a class not found exception, I know this class is not already loaded and so I look up a local repository to fetch a string based script and call parseClass (I do all of this in the context of loadClass)
>
> This used to work with 2.5.x but when I upgrade to 3.0.5, it sits in a blocked state (not sure if it is blocked on some mutex because of a recursion some where). When I use my debugger , when it gets to this state, I am not able to read any of my variable values in the debugger (which seems to indicate some threading read lock somewhere)
>
> Question :- Am I doing this the right way? Why does it block and not load the class? It locks up in the parseClass invocation (inside a validate call that validates the sourceCode object
>
> My code looks like this
>
> class MyClassLoader constructor(
>     private val repository: FlowsRepository,
>     private val configuration: CompilerConfiguration,
>     parentClassLoader: ClassLoader
> ) : GroovyClassLoader(parentClassLoader, configuration) {
>
>     /** Load class [name] either from repository or class path */
>     override fun loadClass(
>         name: String,
>         lookupScriptFiles: Boolean,
>         preferClassOverScript: Boolean,
>         resolve: Boolean
>     ): Class<*> {
>         return try {
>             // Never try to load from script files, always parse from repository
>             super.loadClass(name, false, true, resolve)
>         } catch (e: ClassNotFoundException) {
>             // Try fetching it from the repository
>             parseFromRepository(name)
>         }
>     }
>
>     /** Parse [name] from the flow repository and cache it */
>     private fun parseFromRepository(name: String): Class<*> {
>
>         // Fetch flow source
>         val flow = repository.fetchFlow(name) ?: throw ClassNotFoundException("Unable to find flow $name")
>
>         // Parse the class if found
>         try {
>             // If parseClass yields a class whose name does not match (typically when we are trying to find an inner
>             // class (A.B) by parsing the outer class source file (A.groovy), throw class not found. The compiler will
>             // attempt to then find A$B which is how inner classes are stored in the class loader. This find will
>             // succeed (this process is repeated for inner classes inside inner classes)
>             super.parseClass(flow.sourceCode, true)
>
>             // The class parsed may not be the class you are looking for, but it may be an inner class, so
>             // use the cache to get it
>             return getClassCacheEntry(name) ?: throw ClassNotFoundException("Unable to find flow $name")
>         } catch (e: CompilationFailedException) {
>             throw CodeQualityException("Compile error", e)
>         }
>     }
> }
>
Apache Groovy committer & PMC member

Blog: http://blog.sunlan.me
Twitter: @daniel_sun
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading from 2.5 to 3.0.5 and my class loader extension does not work

Saravanan Palanichamy
Apologies to the group

This was a bad dependency merge of junit that was causing this thread lockup. It works just fine now

regards
Saravanan

On 2020/08/10 04:41:42, Daniel Sun <[hidden email]> wrote:

> Could you provide a standalone runnable script to reproduce the issue?
> BTW, the script is expected to be written in Groovy/Java.
>
> Cheers,
> Daniel Sun
> On 2020/08/08 13:19:01, Saravanan Palanichamy <[hidden email]> wrote:
> > Hello
> >
> > I have an extension of GroovyClassLoader that I use as follows
> >
> > a) I override loadClass to make sure I decide where to load from
> > b) I first call loadClass on my parent class loader to make sure the class is not loaded there (I assert if it is. This code is not shown below)
> > c) I then call loadClass after turning off all script loading on myclassloader to make sure this class is not already loaded. If it is, I just use that
> > d) If I get a class not found exception, I know this class is not already loaded and so I look up a local repository to fetch a string based script and call parseClass (I do all of this in the context of loadClass)
> >
> > This used to work with 2.5.x but when I upgrade to 3.0.5, it sits in a blocked state (not sure if it is blocked on some mutex because of a recursion some where). When I use my debugger , when it gets to this state, I am not able to read any of my variable values in the debugger (which seems to indicate some threading read lock somewhere)
> >
> > Question :- Am I doing this the right way? Why does it block and not load the class? It locks up in the parseClass invocation (inside a validate call that validates the sourceCode object
> >
> > My code looks like this
> >
> > class MyClassLoader constructor(
> >     private val repository: FlowsRepository,
> >     private val configuration: CompilerConfiguration,
> >     parentClassLoader: ClassLoader
> > ) : GroovyClassLoader(parentClassLoader, configuration) {
> >
> >     /** Load class [name] either from repository or class path */
> >     override fun loadClass(
> >         name: String,
> >         lookupScriptFiles: Boolean,
> >         preferClassOverScript: Boolean,
> >         resolve: Boolean
> >     ): Class<*> {
> >         return try {
> >             // Never try to load from script files, always parse from repository
> >             super.loadClass(name, false, true, resolve)
> >         } catch (e: ClassNotFoundException) {
> >             // Try fetching it from the repository
> >             parseFromRepository(name)
> >         }
> >     }
> >
> >     /** Parse [name] from the flow repository and cache it */
> >     private fun parseFromRepository(name: String): Class<*> {
> >
> >         // Fetch flow source
> >         val flow = repository.fetchFlow(name) ?: throw ClassNotFoundException("Unable to find flow $name")
> >
> >         // Parse the class if found
> >         try {
> >             // If parseClass yields a class whose name does not match (typically when we are trying to find an inner
> >             // class (A.B) by parsing the outer class source file (A.groovy), throw class not found. The compiler will
> >             // attempt to then find A$B which is how inner classes are stored in the class loader. This find will
> >             // succeed (this process is repeated for inner classes inside inner classes)
> >             super.parseClass(flow.sourceCode, true)
> >
> >             // The class parsed may not be the class you are looking for, but it may be an inner class, so
> >             // use the cache to get it
> >             return getClassCacheEntry(name) ?: throw ClassNotFoundException("Unable to find flow $name")
> >         } catch (e: CompilationFailedException) {
> >             throw CodeQualityException("Compile error", e)
> >         }
> >     }
> > }
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading from 2.5 to 3.0.5 and my class loader extension does not work

Saravanan Palanichamy
In reply to this post by Daniel.Sun
Hello Daniel

I was able to reproduce this error. I am still not sure if it is my setup that is causing this error. I have attached my code files as a zip file

My class loader code is this. Notice that I overload the loadClass method and if I hit a class not found exception, I try to parse the class through a second parseClass method.

The problem I see is this. I am importing another class (Pojo) from the first groovy file. So when I compile the first file, internally it tries to resolve Pojo and tries to compile the second file. Which is where it hits the getAndPut call in sourceCache.

This works in 2.5 because it is using a concurrentcommoncache for sourceCache, whereas 3.0.5 is using a StampedCommonCache which does not allow recursion on getAndPut. The logs I see in my console show I am attempting to load the top most class and its import class one after the other. My unit test is also added towards the end of this file

public class MyClassLoader extends GroovyClassLoader {

private FlowRepository repository;

private CompilerConfiguration configuration;

public MyClassLoader(ClassLoader parentClassLoader, CompilerConfiguration configuration, FlowRepository repository) {
super(parentClassLoader, configuration);
this.repository = repository;
}

@Override
public Class loadClass(
String name,
boolean lookupScriptFiles,
boolean preferClassOverScript,
boolean resolve
) throws ClassNotFoundException, CompilationFailedException {

try {
// Never try to load from script files, always parse from repository
return super.loadClass(name, false, true, resolve);
} catch (ClassNotFoundException e) {
// Try fetching it from the repository
return parseFromRepository(name);
}
}

/**
* Parse [name] from the flow repository and cache it
*/
private Class parseFromRepository(String name) throws UnsupportedOperationException, ClassNotFoundException {

// Fetch flow source
Flow flow = repository.fetchFlow(name);
if (flow == null)
throw new ClassNotFoundException("Unable to find flow $name");

// Parse the class if found
try {

System.out.println("Loading class " + flow.sourceCode.getName());

// If parseClass yields a class whose name does not match (typically when we are trying to find an inner
// class (A.B) by parsing the outer class source file (A.groovy), throw class not found. The compiler will
// attempt to then find A$B which is how inner classes are stored in the class loader. This find will
// succeed (this process is repeated for inner classes inside inner classes)
super.parseClass(flow.sourceCode, true);

System.out.println("Finished loading class " + flow.sourceCode.getName());

// The class parsed may not be the class you are looking for, but it may be an inner class, so
// use the cache to get it
Class cachedClass = super.getClassCacheEntry(name);
if (cachedClass != null)
return cachedClass;
else
throw new ClassNotFoundException("Unable to find flow $name");
} catch (CompilationFailedException e) {
throw new UnsupportedOperationException("Compile error", e);
}
}
}

@Test
fun testMe() {
val repository = FlowFolderRepository()
val config = CompilerConfiguration().addCompilationCustomizers(
ASTTransformationCustomizer(CompileStatic::class.java)
)
val sandbox = MyClassLoader(this::class.java.classLoader, config, repository)

// Load all flows in the repository
for (flow in repository.fetchAllFlows()) {

// Make sure class is not already loaded in the parent class loader. This is usually a Config file error
try {
this::class.java.classLoader.loadClass(flow.name)
throw ConfigQualityException("${flow.name} is already loaded and cannot be recompiled.")
} catch (e: ClassNotFoundException) {
}
        // Load the class
        sandbox.loadClass(flow.name)
}
}

On Mon, Aug 10, 2020 at 10:11 AM Daniel Sun <[hidden email]> wrote:
Could you provide a standalone runnable script to reproduce the issue?
BTW, the script is expected to be written in Groovy/Java.

Cheers,
Daniel Sun
On 2020/08/08 13:19:01, Saravanan Palanichamy <[hidden email]> wrote:
> Hello
>
> I have an extension of GroovyClassLoader that I use as follows
>
> a) I override loadClass to make sure I decide where to load from
> b) I first call loadClass on my parent class loader to make sure the class is not loaded there (I assert if it is. This code is not shown below)
> c) I then call loadClass after turning off all script loading on myclassloader to make sure this class is not already loaded. If it is, I just use that
> d) If I get a class not found exception, I know this class is not already loaded and so I look up a local repository to fetch a string based script and call parseClass (I do all of this in the context of loadClass)
>
> This used to work with 2.5.x but when I upgrade to 3.0.5, it sits in a blocked state (not sure if it is blocked on some mutex because of a recursion some where). When I use my debugger , when it gets to this state, I am not able to read any of my variable values in the debugger (which seems to indicate some threading read lock somewhere)
>
> Question :- Am I doing this the right way? Why does it block and not load the class? It locks up in the parseClass invocation (inside a validate call that validates the sourceCode object
>
> My code looks like this
>
> class MyClassLoader constructor(
>     private val repository: FlowsRepository,
>     private val configuration: CompilerConfiguration,
>     parentClassLoader: ClassLoader
> ) : GroovyClassLoader(parentClassLoader, configuration) {
>
>     /** Load class [name] either from repository or class path */
>     override fun loadClass(
>         name: String,
>         lookupScriptFiles: Boolean,
>         preferClassOverScript: Boolean,
>         resolve: Boolean
>     ): Class<*> {
>         return try {
>             // Never try to load from script files, always parse from repository
>             super.loadClass(name, false, true, resolve)
>         } catch (e: ClassNotFoundException) {
>             // Try fetching it from the repository
>             parseFromRepository(name)
>         }
>     }
>
>     /** Parse [name] from the flow repository and cache it */
>     private fun parseFromRepository(name: String): Class<*> {
>
>         // Fetch flow source
>         val flow = repository.fetchFlow(name) ?: throw ClassNotFoundException("Unable to find flow $name")
>
>         // Parse the class if found
>         try {
>             // If parseClass yields a class whose name does not match (typically when we are trying to find an inner
>             // class (A.B) by parsing the outer class source file (A.groovy), throw class not found. The compiler will
>             // attempt to then find A$B which is how inner classes are stored in the class loader. This find will
>             // succeed (this process is repeated for inner classes inside inner classes)
>             super.parseClass(flow.sourceCode, true)
>
>             // The class parsed may not be the class you are looking for, but it may be an inner class, so
>             // use the cache to get it
>             return getClassCacheEntry(name) ?: throw ClassNotFoundException("Unable to find flow $name")
>         } catch (e: CompilationFailedException) {
>             throw CodeQualityException("Compile error", e)
>         }
>     }
> }
>

ClassLoad.tgz (13K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading from 2.5 to 3.0.5 and my class loader extension does not work

Saravanan Palanichamy
While debugging this, I noticed this. Intellij tells me that the source file and class file dont match. The debugging also has a weird jump from GroovyClassLoader::doParseClass() line number 366 to StampedCommonCache::isEmpty. I am using JAVA 11 and Groovy 3.0.5

Screenshot 2020-08-17 at 11.28.44 AM.png (293K) Download Attachment
Screenshot 2020-08-17 at 11.28.09 AM.png (250K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading from 2.5 to 3.0.5 and my class loader extension does not work

Saravanan Palanichamy
Digging into this further, It looks like StampedCommonCache is not reentrant. But in my case, I am calling loadClass from inside another loadClass. Is this the problem? What is the recommendation to compile a set of classes together that have dependencies between each other?

In the attachment, notice how loadClass is called within loadClass. This is because one of my import statements in the first class requires a second class load

regards
Saravanan

On Mon, Aug 17, 2020 at 11:33 AM Saravanan Palanichamy <[hidden email]> wrote:
While debugging this, I noticed this. Intellij tells me that the source file and class file dont match. The debugging also has a weird jump from GroovyClassLoader::doParseClass() line number 366 to StampedCommonCache::isEmpty. I am using JAVA 11 and Groovy 3.0.5

Screenshot 2020-08-17 at 3.29.58 PM.png (999K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading from 2.5 to 3.0.5 and my class loader extension does not work

Daniel.Sun
Hi Saravanan,

       Sorry for the late reply... I am too busy recently...

       Yep, `StampedCommonCache` is not reentrant. You could try to replace `StampedCommonCache ` with `org.codehaus.groovy.runtime.memoize.UnlimitedConcurrentCache`, and build Groovy via `gradlew clean installGroovy --no-build-cache` to get the binary distribution.

       If it works, PR is welcomed. Good luck ;-)

Cheers,
Daniel Sun

On 2020/08/17 10:02:12, Saravanan Palanichamy <[hidden email]> wrote:

> Digging into this further, It looks like StampedCommonCache is not
> reentrant. But in my case, I am calling loadClass from inside another
> loadClass. Is this the problem? What is the recommendation to compile a set
> of classes together that have dependencies between each other?
>
> In the attachment, notice how loadClass is called within loadClass. This is
> because one of my import statements in the first class requires a second
> class load
>
> regards
> Saravanan
>
> On Mon, Aug 17, 2020 at 11:33 AM Saravanan Palanichamy <[hidden email]>
> wrote:
>
> > While debugging this, I noticed this. Intellij tells me that the source
> > file and class file dont match. The debugging also has a weird jump from
> > GroovyClassLoader::doParseClass() line number 366 to
> > StampedCommonCache::isEmpty. I am using JAVA 11 and Groovy 3.0.5
> >
>
Apache Groovy committer & PMC member

Blog: http://blog.sunlan.me
Twitter: @daniel_sun