Quantcast

Trying to debug a permgen leak... what is this Class?

classic Classic list List threaded Threaded
68 messages Options
1234567
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Jochen Theodorou
Am 21.01.2011 18:20, schrieb Alex Tkachman:

> ClassInfo has
>
>      private final LazyClassLoaderRef artifactClassLoader;
>
>      private static class LazyClassLoaderRef extends
> LazyReference<ClassLoaderForClassArtifacts>  {
>          private final ClassInfo info;
>
>          LazyClassLoaderRef(ReferenceBundle bundle, ClassInfo info) {
>              super(bundle);
>              this.info = info;
>          }
>
>          public ClassLoaderForClassArtifacts initValue() {
>              return new ClassLoaderForClassArtifacts(info.get());
>          }
>      }

That LazyRef means, afaik, that it is SoftReferenced. If garbage
collection of this loader is not possible, then I see these possible
reasons for it:

* the loader is referenced somewhere else too
* a class defined by that loader is more than soft reachable
* an object of a class of that loader is
* a gc bug

the question is, how we can get that information

bye blackdrag

--
Jochen "blackdrag" Theodorou
The Groovy Project Tech Lead
http://blackdragsview.blogspot.com/
For Groovy programming sources visit http://groovy.codehaus.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Alex Tkachman
In reply to this post by Marc Palmer Local
Marc,

Most likely absolutely irrelevant but do you have chances to see if
following static fields are both NULL GroovySunClassLoader.sunVM and
SunClassLoader.sunVM?

Alex

On Wed, Jan 19, 2011 at 5:36 PM, Marc Palmer <[hidden email]> wrote:

> Hi,
>
> Some of us are experiencing permgen leaks on grails app in production (Weceem CMS). However I cannot reproduce this under heavy simulated load locally, no matter what I do. It takes a few weaks to blow permgen in production.
>
> I do however have a heapdump from the production VM that shows that we have 99,000+ classes loaded. So we have a class leak.
>
> Our code does do some loading of dynamic Grails GSP content, but like I said this is not reproducible locally, that same code does not leak over 10,000s of requests - the GSP classes that are "reloaded" are GC'd from permgen, I have verified this with visualvm.
>
> Now, inspecting the heapdump I see a *lot* of classes (1000s) that have no instances, of the form:
>
> org.weceem.services.WcmContentRepositoryService$publishPendingContent$2497
>
> What is this class? It's not a closure is it, as there is no _closure in the name. The "publishPendingContent" method, for example, does have some closures in it, and there are classes for those:
>
> org.weceem.services.WcmContentRepositoryService$_publishPendingContent_closure53
> org.weceem.services.WcmContentRepositoryService$_publishPendingContent_closure53_closure70
> org.weceem.services.WcmContentRepositoryService$_publishPendingContent_closure53_closure70_closure71
>
> ...however I have nearly 3000 classes of the publishPendingContent$NNNN form, from:
>
> org.weceem.services.WcmContentRepositoryService$publishPendingContent$2839
>
> to
>
> org.weceem.services.WcmContentRepositoryService$publishPendingContent$2839
>
>
> There are no instances of any of them and none of them are gc'ing. We have thousands of these for the methods of our grails service code.
>
> It is as if each execution (in some cases) of these methods is causing a new class to be generated, and these are not gc'able.
>
> As a result, after a week or two permgen blows up. This is with the version of Groovy shipped with Grails 1.3.6.
>
> A bit of investigation shows that Class(es) of this naming convention appear to be referenced in the returnType field of Groovy's CachedMethod. Does this sound correct?
>
> Why are these being created in their thousands? And also, why do the CacheMethods not GC? Is this because normal heap is not being exhausted, so cached methods are not gc'd and so the refs to these ad-hoc classes cannot be gc'd by permgen?
>
> Thanks in advance
> Marc
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Alex Tkachman
In reply to this post by Jochen Theodorou
I am almost sure now (after inspecting code) that class loader is only
soft referenced

On Fri, Jan 21, 2011 at 8:39 PM, Jochen Theodorou <[hidden email]> wrote:

> Am 21.01.2011 18:20, schrieb Alex Tkachman:
>>
>> ClassInfo has
>>
>>     private final LazyClassLoaderRef artifactClassLoader;
>>
>>     private static class LazyClassLoaderRef extends
>> LazyReference<ClassLoaderForClassArtifacts>  {
>>         private final ClassInfo info;
>>
>>         LazyClassLoaderRef(ReferenceBundle bundle, ClassInfo info) {
>>             super(bundle);
>>             this.info = info;
>>         }
>>
>>         public ClassLoaderForClassArtifacts initValue() {
>>             return new ClassLoaderForClassArtifacts(info.get());
>>         }
>>     }
>
> That LazyRef means, afaik, that it is SoftReferenced. If garbage collection
> of this loader is not possible, then I see these possible reasons for it:
>
> * the loader is referenced somewhere else too
> * a class defined by that loader is more than soft reachable
> * an object of a class of that loader is
> * a gc bug
>
> the question is, how we can get that information
>
> bye blackdrag
>
> --
> Jochen "blackdrag" Theodorou
> The Groovy Project Tech Lead
> http://blackdragsview.blogspot.com/
> For Groovy programming sources visit http://groovy.codehaus.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>   http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

melix
In reply to this post by Jochen Theodorou
VisualVM is IMHO too limited to diagnose such a thing. You could try to
download the 15-day evaluation version of YourKit Java profiler. It has
a very handy feature for those cases : pathes from GC roots. It will
allow you to find where those classes are referenced.

I use this tool quite often, it's really useful for that kind of problems.

> That LazyRef means, afaik, that it is SoftReferenced. If garbage
> collection of this loader is not possible, then I see these possible
> reasons for it:
>
> * the loader is referenced somewhere else too
> * a class defined by that loader is more than soft reachable
> * an object of a class of that loader is
> * a gc bug
>
> the question is, how we can get that information
>
> bye blackdrag
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Alex Tkachman
In fact we don't need eval version. I think YourKIt guys provides
Codehaus projects with the license

On Fri, Jan 21, 2011 at 9:04 PM, Cédric CHAMPEAU
<[hidden email]> wrote:

> VisualVM is IMHO too limited to diagnose such a thing. You could try to
> download the 15-day evaluation version of YourKit Java profiler. It has a
> very handy feature for those cases : pathes from GC roots. It will allow you
> to find where those classes are referenced.
>
> I use this tool quite often, it's really useful for that kind of problems.
>
>> That LazyRef means, afaik, that it is SoftReferenced. If garbage
>> collection of this loader is not possible, then I see these possible reasons
>> for it:
>>
>> * the loader is referenced somewhere else too
>> * a class defined by that loader is more than soft reachable
>> * an object of a class of that loader is
>> * a gc bug
>>
>> the question is, how we can get that information
>>
>> bye blackdrag
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>   http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Marc Palmer Local

On 21 Jan 2011, at 20:08, Alex Tkachman wrote:

> In fact we don't need eval version. I think YourKIt guys provides
> Codehaus projects with the license

Not any more it seems. My Xircles account only shows JProfiler license.

I'm going to try with an eval license. FWIW I think jvisualvm can do what we need, *if* you can work out the right OQL incantation to analyse the gc roots.

Marc


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Marc Palmer Local
In reply to this post by Alex Tkachman

OK so YourKit is very buggy on OS X and it is difficult to get info out of it, as it is not showing classes that are definitely shown in jvisualvm.

However I found two references to instances of WcmContentRepositoryService$getGSPTemplate$2020 shown here:

https://skitch.com/marcpalmer/rk5xp/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

There are a *lot* of classloaders too.

Here's the last of thee 1359 classes for CharSequence.length:

https://skitch.com/marcpalmer/rk5tw/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

This shows class 1358 (not the last one):

https://skitch.com/marcpalmer/rk51e/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

Is this the evidence we need? IT shows refs from thread local CallSite array. The two CallSiteClassLoader(s) listed there are not the same class loader instance.

I looked at the first callsite for WcmContentRepositoryServic$archiveStaleContent (no number appended) and this gives different, interesting results:

https://skitch.com/marcpalmer/rk5uu/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

This shows that the class is still hard referenced in the vector used by CallSiteClassLoader. The Classloader and vector are still in memory because of newer callsites added later (getTemplateForContent$1934)

So it appears the CallSiteClassLoader hard references the classes, and some failure that is causing multiple call sites to be created (due to a hash miss?) results in the original callsite vector of classes growing and growing, and never releasing the dead classes.

This is the same for e.g. WcmContentRepositoryServic$archiveStaleContent$0 which is also in the same callsite[] retained by getTemplateForContent$1934's classloader:

https://skitch.com/marcpalmer/rk5w5/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

In my view, the root cause is the "hash miss" on the callsite in the first place though. Don't you think? Yes it would be nice if permgen GC was able to recover this stuff, but this problem should never be happening, and we'd still see bad performance even if the gc issue is fixed.

Why is the callsite being regenerated all the time?

Marc





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Jochen Theodorou
Am 24.01.2011 14:34, schrieb Marc Palmer:
[...]
>  This shows that the class is still hard referenced in the vector
> used by CallSiteClassLoader. The Classloader and vector are still in
> memory because of newer callsites added later
> (getTemplateForContent$1934)
>
> So it appears the CallSiteClassLoader hard references the classes,
> and some failure that is causing multiple call sites to be created
> (due to a hash miss?) results in the original callsite vector of
> classes growing and growing, and never releasing the dead classes.

I think here is a slight misunderstanding. A ClassLoader is *always*
supposed to hard reference the classes it defined. That is because
according to the class loading constraints a classloader should always
return the same class if you ask for a certain name. Also the Class
always hardreferences its defining loader.
For an object to be able to be unloaded the key is the reachability. If
A hard refers B and C weakly refers A, then not only is A weak
referenced, so is also B, but B is still only collected either together
with A or together with A. If D hard references B, then it is changed in
that B is now no longer weak reachable, but A still is. So in that case
A could be collected, but not B.
Since Class and defining ClassLoader hardreference each other it is
essential for unloading, that neither the Classes nor the defining
ClassLoader is more then soft/weak reachable.

If I interpret
https://skitch.com/marcpalmer/rk5tw/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4 
correctly, then it tells me that the Class
java_lang_CharSequence$length$1359 is Weak/Soft reachable, meaning that
the loader is as well. That means this Class could be collected, but
obviously is not!

bye blackdrag

--
Jochen "blackdrag" Theodorou
The Groovy Project Tech Lead
http://blackdragsview.blogspot.com/
For Groovy programming sources visit http://groovy.codehaus.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Alex Tkachman
I am under the same impression.

The interesting fact I read from the same screenshot is that it is
proven now we generates tons of classes for the same method.

Jochen, could you please have a look on CachedMethod and how we cache
constructors of generated classes. I start thinking that may be we
deal with classical concurrency issue on Java Memory Model.

On Mon, Jan 24, 2011 at 3:09 PM, Jochen Theodorou <[hidden email]> wrote:

> Am 24.01.2011 14:34, schrieb Marc Palmer:
> [...]
>>
>>  This shows that the class is still hard referenced in the vector
>> used by CallSiteClassLoader. The Classloader and vector are still in
>> memory because of newer callsites added later
>> (getTemplateForContent$1934)
>>
>> So it appears the CallSiteClassLoader hard references the classes,
>> and some failure that is causing multiple call sites to be created
>> (due to a hash miss?) results in the original callsite vector of
>> classes growing and growing, and never releasing the dead classes.
>
> I think here is a slight misunderstanding. A ClassLoader is *always*
> supposed to hard reference the classes it defined. That is because according
> to the class loading constraints a classloader should always return the same
> class if you ask for a certain name. Also the Class always hardreferences
> its defining loader.
> For an object to be able to be unloaded the key is the reachability. If A
> hard refers B and C weakly refers A, then not only is A weak referenced, so
> is also B, but B is still only collected either together with A or together
> with A. If D hard references B, then it is changed in that B is now no
> longer weak reachable, but A still is. So in that case A could be collected,
> but not B.
> Since Class and defining ClassLoader hardreference each other it is
> essential for unloading, that neither the Classes nor the defining
> ClassLoader is more then soft/weak reachable.
>
> If I interpret
> https://skitch.com/marcpalmer/rk5tw/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4
> correctly, then it tells me that the Class
> java_lang_CharSequence$length$1359 is Weak/Soft reachable, meaning that the
> loader is as well. That means this Class could be collected, but obviously
> is not!
>
> bye blackdrag
>
> --
> Jochen "blackdrag" Theodorou
> The Groovy Project Tech Lead
> http://blackdragsview.blogspot.com/
> For Groovy programming sources visit http://groovy.codehaus.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>   http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Marc Palmer Local

On 24 Jan 2011, at 14:16, Alex Tkachman wrote:

> I am under the same impression.
>
> The interesting fact I read from the same screenshot is that it is
> proven now we generates tons of classes for the same method.
>
> Jochen, could you please have a look on CachedMethod and how we cache
> constructors of generated classes. I start thinking that may be we
> deal with classical concurrency issue on Java Memory Model.
>

OK, I'm just wondering how it was not proven that we had 1000s of classes. I have said this all along

In fact I can see that each Class that has callsites generated, has its own groovy CallSiteClassLoader, and these classloader typically (in this heapdump) but not always have 1000s of classes in.

1000s per method on our service WcmContentRepositoryService:

https://skitch.com/marcpalmer/rkh8y/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

Many 1000s per method on java.util.Map:

https://skitch.com/marcpalmer/rkh8y/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

Over 3000 on groovy.util.ConfigObject:

https://skitch.com/marcpalmer/rkh82/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

It is interesting that I have never seen this myself on my own MBP running Apple Java 6 in 64 bit mode.

The server, as previously mentioned, is AMD 64 bit, Sun Java 1.6

I am not the only person seeing this in production though. Göran Ehrsson is also seeing it. Other probably are too, but just cycling their boxes every week or two (or don't notice due to redeploys).

Also recall that a Grails 1.3.1 app on the *same server* does not have this problem. So it would seem unlikely to be a JMM issue?

Marc







---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


1234567
Loading...