Quantcast

Trying to debug a permgen leak... what is this Class?

classic Classic list List threaded Threaded
68 messages Options
1234
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Jochen Theodorou
Am 21.01.2011 18:20, schrieb Alex Tkachman:

> ClassInfo has
>
>      private final LazyClassLoaderRef artifactClassLoader;
>
>      private static class LazyClassLoaderRef extends
> LazyReference<ClassLoaderForClassArtifacts>  {
>          private final ClassInfo info;
>
>          LazyClassLoaderRef(ReferenceBundle bundle, ClassInfo info) {
>              super(bundle);
>              this.info = info;
>          }
>
>          public ClassLoaderForClassArtifacts initValue() {
>              return new ClassLoaderForClassArtifacts(info.get());
>          }
>      }

That LazyRef means, afaik, that it is SoftReferenced. If garbage
collection of this loader is not possible, then I see these possible
reasons for it:

* the loader is referenced somewhere else too
* a class defined by that loader is more than soft reachable
* an object of a class of that loader is
* a gc bug

the question is, how we can get that information

bye blackdrag

--
Jochen "blackdrag" Theodorou
The Groovy Project Tech Lead
http://blackdragsview.blogspot.com/
For Groovy programming sources visit http://groovy.codehaus.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Alex Tkachman
In reply to this post by Marc Palmer Local
Marc,

Most likely absolutely irrelevant but do you have chances to see if
following static fields are both NULL GroovySunClassLoader.sunVM and
SunClassLoader.sunVM?

Alex

On Wed, Jan 19, 2011 at 5:36 PM, Marc Palmer <[hidden email]> wrote:

> Hi,
>
> Some of us are experiencing permgen leaks on grails app in production (Weceem CMS). However I cannot reproduce this under heavy simulated load locally, no matter what I do. It takes a few weaks to blow permgen in production.
>
> I do however have a heapdump from the production VM that shows that we have 99,000+ classes loaded. So we have a class leak.
>
> Our code does do some loading of dynamic Grails GSP content, but like I said this is not reproducible locally, that same code does not leak over 10,000s of requests - the GSP classes that are "reloaded" are GC'd from permgen, I have verified this with visualvm.
>
> Now, inspecting the heapdump I see a *lot* of classes (1000s) that have no instances, of the form:
>
> org.weceem.services.WcmContentRepositoryService$publishPendingContent$2497
>
> What is this class? It's not a closure is it, as there is no _closure in the name. The "publishPendingContent" method, for example, does have some closures in it, and there are classes for those:
>
> org.weceem.services.WcmContentRepositoryService$_publishPendingContent_closure53
> org.weceem.services.WcmContentRepositoryService$_publishPendingContent_closure53_closure70
> org.weceem.services.WcmContentRepositoryService$_publishPendingContent_closure53_closure70_closure71
>
> ...however I have nearly 3000 classes of the publishPendingContent$NNNN form, from:
>
> org.weceem.services.WcmContentRepositoryService$publishPendingContent$2839
>
> to
>
> org.weceem.services.WcmContentRepositoryService$publishPendingContent$2839
>
>
> There are no instances of any of them and none of them are gc'ing. We have thousands of these for the methods of our grails service code.
>
> It is as if each execution (in some cases) of these methods is causing a new class to be generated, and these are not gc'able.
>
> As a result, after a week or two permgen blows up. This is with the version of Groovy shipped with Grails 1.3.6.
>
> A bit of investigation shows that Class(es) of this naming convention appear to be referenced in the returnType field of Groovy's CachedMethod. Does this sound correct?
>
> Why are these being created in their thousands? And also, why do the CacheMethods not GC? Is this because normal heap is not being exhausted, so cached methods are not gc'd and so the refs to these ad-hoc classes cannot be gc'd by permgen?
>
> Thanks in advance
> Marc
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Alex Tkachman
In reply to this post by Jochen Theodorou
I am almost sure now (after inspecting code) that class loader is only
soft referenced

On Fri, Jan 21, 2011 at 8:39 PM, Jochen Theodorou <[hidden email]> wrote:

> Am 21.01.2011 18:20, schrieb Alex Tkachman:
>>
>> ClassInfo has
>>
>>     private final LazyClassLoaderRef artifactClassLoader;
>>
>>     private static class LazyClassLoaderRef extends
>> LazyReference<ClassLoaderForClassArtifacts>  {
>>         private final ClassInfo info;
>>
>>         LazyClassLoaderRef(ReferenceBundle bundle, ClassInfo info) {
>>             super(bundle);
>>             this.info = info;
>>         }
>>
>>         public ClassLoaderForClassArtifacts initValue() {
>>             return new ClassLoaderForClassArtifacts(info.get());
>>         }
>>     }
>
> That LazyRef means, afaik, that it is SoftReferenced. If garbage collection
> of this loader is not possible, then I see these possible reasons for it:
>
> * the loader is referenced somewhere else too
> * a class defined by that loader is more than soft reachable
> * an object of a class of that loader is
> * a gc bug
>
> the question is, how we can get that information
>
> bye blackdrag
>
> --
> Jochen "blackdrag" Theodorou
> The Groovy Project Tech Lead
> http://blackdragsview.blogspot.com/
> For Groovy programming sources visit http://groovy.codehaus.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>   http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

melix
In reply to this post by Jochen Theodorou
VisualVM is IMHO too limited to diagnose such a thing. You could try to
download the 15-day evaluation version of YourKit Java profiler. It has
a very handy feature for those cases : pathes from GC roots. It will
allow you to find where those classes are referenced.

I use this tool quite often, it's really useful for that kind of problems.

> That LazyRef means, afaik, that it is SoftReferenced. If garbage
> collection of this loader is not possible, then I see these possible
> reasons for it:
>
> * the loader is referenced somewhere else too
> * a class defined by that loader is more than soft reachable
> * an object of a class of that loader is
> * a gc bug
>
> the question is, how we can get that information
>
> bye blackdrag
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Alex Tkachman
In fact we don't need eval version. I think YourKIt guys provides
Codehaus projects with the license

On Fri, Jan 21, 2011 at 9:04 PM, Cédric CHAMPEAU
<[hidden email]> wrote:

> VisualVM is IMHO too limited to diagnose such a thing. You could try to
> download the 15-day evaluation version of YourKit Java profiler. It has a
> very handy feature for those cases : pathes from GC roots. It will allow you
> to find where those classes are referenced.
>
> I use this tool quite often, it's really useful for that kind of problems.
>
>> That LazyRef means, afaik, that it is SoftReferenced. If garbage
>> collection of this loader is not possible, then I see these possible reasons
>> for it:
>>
>> * the loader is referenced somewhere else too
>> * a class defined by that loader is more than soft reachable
>> * an object of a class of that loader is
>> * a gc bug
>>
>> the question is, how we can get that information
>>
>> bye blackdrag
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>   http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Marc Palmer Local

On 21 Jan 2011, at 20:08, Alex Tkachman wrote:

> In fact we don't need eval version. I think YourKIt guys provides
> Codehaus projects with the license

Not any more it seems. My Xircles account only shows JProfiler license.

I'm going to try with an eval license. FWIW I think jvisualvm can do what we need, *if* you can work out the right OQL incantation to analyse the gc roots.

Marc


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Marc Palmer Local
In reply to this post by Alex Tkachman

OK so YourKit is very buggy on OS X and it is difficult to get info out of it, as it is not showing classes that are definitely shown in jvisualvm.

However I found two references to instances of WcmContentRepositoryService$getGSPTemplate$2020 shown here:

https://skitch.com/marcpalmer/rk5xp/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

There are a *lot* of classloaders too.

Here's the last of thee 1359 classes for CharSequence.length:

https://skitch.com/marcpalmer/rk5tw/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

This shows class 1358 (not the last one):

https://skitch.com/marcpalmer/rk51e/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

Is this the evidence we need? IT shows refs from thread local CallSite array. The two CallSiteClassLoader(s) listed there are not the same class loader instance.

I looked at the first callsite for WcmContentRepositoryServic$archiveStaleContent (no number appended) and this gives different, interesting results:

https://skitch.com/marcpalmer/rk5uu/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

This shows that the class is still hard referenced in the vector used by CallSiteClassLoader. The Classloader and vector are still in memory because of newer callsites added later (getTemplateForContent$1934)

So it appears the CallSiteClassLoader hard references the classes, and some failure that is causing multiple call sites to be created (due to a hash miss?) results in the original callsite vector of classes growing and growing, and never releasing the dead classes.

This is the same for e.g. WcmContentRepositoryServic$archiveStaleContent$0 which is also in the same callsite[] retained by getTemplateForContent$1934's classloader:

https://skitch.com/marcpalmer/rk5w5/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

In my view, the root cause is the "hash miss" on the callsite in the first place though. Don't you think? Yes it would be nice if permgen GC was able to recover this stuff, but this problem should never be happening, and we'd still see bad performance even if the gc issue is fixed.

Why is the callsite being regenerated all the time?

Marc





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Jochen Theodorou
Am 24.01.2011 14:34, schrieb Marc Palmer:
[...]
>  This shows that the class is still hard referenced in the vector
> used by CallSiteClassLoader. The Classloader and vector are still in
> memory because of newer callsites added later
> (getTemplateForContent$1934)
>
> So it appears the CallSiteClassLoader hard references the classes,
> and some failure that is causing multiple call sites to be created
> (due to a hash miss?) results in the original callsite vector of
> classes growing and growing, and never releasing the dead classes.

I think here is a slight misunderstanding. A ClassLoader is *always*
supposed to hard reference the classes it defined. That is because
according to the class loading constraints a classloader should always
return the same class if you ask for a certain name. Also the Class
always hardreferences its defining loader.
For an object to be able to be unloaded the key is the reachability. If
A hard refers B and C weakly refers A, then not only is A weak
referenced, so is also B, but B is still only collected either together
with A or together with A. If D hard references B, then it is changed in
that B is now no longer weak reachable, but A still is. So in that case
A could be collected, but not B.
Since Class and defining ClassLoader hardreference each other it is
essential for unloading, that neither the Classes nor the defining
ClassLoader is more then soft/weak reachable.

If I interpret
https://skitch.com/marcpalmer/rk5tw/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4 
correctly, then it tells me that the Class
java_lang_CharSequence$length$1359 is Weak/Soft reachable, meaning that
the loader is as well. That means this Class could be collected, but
obviously is not!

bye blackdrag

--
Jochen "blackdrag" Theodorou
The Groovy Project Tech Lead
http://blackdragsview.blogspot.com/
For Groovy programming sources visit http://groovy.codehaus.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Alex Tkachman
I am under the same impression.

The interesting fact I read from the same screenshot is that it is
proven now we generates tons of classes for the same method.

Jochen, could you please have a look on CachedMethod and how we cache
constructors of generated classes. I start thinking that may be we
deal with classical concurrency issue on Java Memory Model.

On Mon, Jan 24, 2011 at 3:09 PM, Jochen Theodorou <[hidden email]> wrote:

> Am 24.01.2011 14:34, schrieb Marc Palmer:
> [...]
>>
>>  This shows that the class is still hard referenced in the vector
>> used by CallSiteClassLoader. The Classloader and vector are still in
>> memory because of newer callsites added later
>> (getTemplateForContent$1934)
>>
>> So it appears the CallSiteClassLoader hard references the classes,
>> and some failure that is causing multiple call sites to be created
>> (due to a hash miss?) results in the original callsite vector of
>> classes growing and growing, and never releasing the dead classes.
>
> I think here is a slight misunderstanding. A ClassLoader is *always*
> supposed to hard reference the classes it defined. That is because according
> to the class loading constraints a classloader should always return the same
> class if you ask for a certain name. Also the Class always hardreferences
> its defining loader.
> For an object to be able to be unloaded the key is the reachability. If A
> hard refers B and C weakly refers A, then not only is A weak referenced, so
> is also B, but B is still only collected either together with A or together
> with A. If D hard references B, then it is changed in that B is now no
> longer weak reachable, but A still is. So in that case A could be collected,
> but not B.
> Since Class and defining ClassLoader hardreference each other it is
> essential for unloading, that neither the Classes nor the defining
> ClassLoader is more then soft/weak reachable.
>
> If I interpret
> https://skitch.com/marcpalmer/rk5tw/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4
> correctly, then it tells me that the Class
> java_lang_CharSequence$length$1359 is Weak/Soft reachable, meaning that the
> loader is as well. That means this Class could be collected, but obviously
> is not!
>
> bye blackdrag
>
> --
> Jochen "blackdrag" Theodorou
> The Groovy Project Tech Lead
> http://blackdragsview.blogspot.com/
> For Groovy programming sources visit http://groovy.codehaus.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>   http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Marc Palmer Local

On 24 Jan 2011, at 14:16, Alex Tkachman wrote:

> I am under the same impression.
>
> The interesting fact I read from the same screenshot is that it is
> proven now we generates tons of classes for the same method.
>
> Jochen, could you please have a look on CachedMethod and how we cache
> constructors of generated classes. I start thinking that may be we
> deal with classical concurrency issue on Java Memory Model.
>

OK, I'm just wondering how it was not proven that we had 1000s of classes. I have said this all along

In fact I can see that each Class that has callsites generated, has its own groovy CallSiteClassLoader, and these classloader typically (in this heapdump) but not always have 1000s of classes in.

1000s per method on our service WcmContentRepositoryService:

https://skitch.com/marcpalmer/rkh8y/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

Many 1000s per method on java.util.Map:

https://skitch.com/marcpalmer/rkh8y/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

Over 3000 on groovy.util.ConfigObject:

https://skitch.com/marcpalmer/rkh82/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

It is interesting that I have never seen this myself on my own MBP running Apple Java 6 in 64 bit mode.

The server, as previously mentioned, is AMD 64 bit, Sun Java 1.6

I am not the only person seeing this in production though. Göran Ehrsson is also seeing it. Other probably are too, but just cycling their boxes every week or two (or don't notice due to redeploys).

Also recall that a Grails 1.3.1 app on the *same server* does not have this problem. So it would seem unlikely to be a JMM issue?

Marc







---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Marc Palmer Local
In reply to this post by Alex Tkachman
One more screen shot, an interesting one:

https://skitch.com/marcpalmer/rkhjn/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4

This shows that one of the many ConfigObject.flatten callsite classes is in a Classloader referenced strongly by the metaClass of ConfigObject itself.

This circular reference means the CallSiteClassLoader can never be gc'd and as a result the major problem is that new callsites are continuously generated, and all referenced from the same CallSite[].

The callsite clases have a "loader" property that hard references their classloader, so the classloader itself can never be gc'd (not that we want this?), and it is hard referenced from the Apache WebAppClassLoader too.

So while the CallSiteClassLoader may be weak/soft ref'd from CachedClass, it is never going to go away.

Again, the real question is why the continuous generation of the callsites.

If we can theorize a reason for their creation, we can run some tests to try to recreate the problem.

Marc


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Jochen Theodorou
In reply to this post by Marc Palmer Local
Am 24.01.2011 16:39, schrieb Marc Palmer:

>
> On 24 Jan 2011, at 14:16, Alex Tkachman wrote:
>
>> I am under the same impression.
>>
>> The interesting fact I read from the same screenshot is that it is
>> proven now we generates tons of classes for the same method.
>>
>> Jochen, could you please have a look on CachedMethod and how we
>> cache constructors of generated classes. I start thinking that may
>> be we deal with classical concurrency issue on Java Memory Model.
>
> OK, I'm just wondering how it was not proven that we had 1000s of
> classes. I have said this all along

oh, I did see that as proven. Only I, and I think Alex too were assuming
that this problem happens, because the classes are referenced somewhere
somewhere and that the GC should normally be able to collect them

> In fact I can see that each Class that has callsites generated, has
> its own groovy CallSiteClassLoader, and these classloader typically
> (in this heapdump) but not always have 1000s of classes in.

but this is only a result of the problem I think, not really the problem
itself. For me the question is more as of why this loader is kept even
though there is not enough permgen space anymore. If this loader is
really collectable, then GC should have collected it. So this is either
a GC bug, or the loader and the classes are not collectible. And since
we should assume the later the question is where the reference is coming
from.

> 1000s per method on our service WcmContentRepositoryService:
>
> https://skitch.com/marcpalmer/rkh8y/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4
>
>  Many 1000s per method on java.util.Map:
>
> https://skitch.com/marcpalmer/rkh8y/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4
>
>  Over 3000 on groovy.util.ConfigObject:
>
> https://skitch.com/marcpalmer/rkh82/java-pid20804.hprof-users-marc-projects-noticelocal-postmortem-cms-pre-emptive-dump-on-go-slow-yourkit-java-profiler-9.5.4
>
>  It is interesting that I have never seen this myself on my own MBP
> running Apple Java 6 in 64 bit mode.
>
> The server, as previously mentioned, is AMD 64 bit, Sun Java 1.6

that would hint a GC bug... but

> I am not the only person seeing this in production though. Göran
> Ehrsson is also seeing it. Other probably are too, but just cycling
> their boxes every week or two (or don't notice due to redeploys).
>
> Also recall that a Grails 1.3.1 app on the *same server* does not
> have this problem. So it would seem unlikely to be a JMM issue?

running the same app with the same JVM and only Grails is different does
not sound like anything that is really related to a GC bug again.

One interesting fact. The name of that call site method class is not the
name the call site is used in, but the name of the method we call. If
you have 3000 places calling flatten, then you get 3000 call site
methods for that flatten call actually. And of course that means, that
these cannot be collected before these places are all collectible as well.

bye blackdrag

--
Jochen "blackdrag" Theodorou
The Groovy Project Tech Lead
http://blackdragsview.blogspot.com/
For Groovy programming sources visit http://groovy.codehaus.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Jochen Theodorou
In reply to this post by Alex Tkachman
Am 24.01.2011 15:16, schrieb Alex Tkachman:
> I am under the same impression.
>
> The interesting fact I read from the same screenshot is that it is
> proven now we generates tons of classes for the same method.

as I said to Marc, that itself is not necessarily wrong. If 3000 places
call ConfigSlurper#flatten, you get 3000 call sites that each has its
own generated method to call ConfigSlurper#flatten.

> Jochen, could you please have a look on CachedMethod and how we cache
> constructors of generated classes. I start thinking that may be we
> deal with classical concurrency issue on Java Memory Model.

I am wondering in a different direction atm actually. The Constructor is
a SoftReference. But what keeps the Constructor from being collected? If
the constructor is not there, then we will have to recreate the method
all the time.

As for JMM... I doubt this is really a problem. The problem mentioned
here appears after weeks of running the application. Even if each thread
had its own variant, it would not lead to such problems. And a partially
initiated SoftReference should cause other problems. A partially
initiated Constructor too.

bye blackdrag

--
Jochen "blackdrag" Theodorou
The Groovy Project Tech Lead
http://blackdragsview.blogspot.com/
For Groovy programming sources visit http://groovy.codehaus.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Alex Tkachman
As I mentioned before Constructor collecting is my first theory. I was
under assumption that being on SoftReference means exactly that it is
not collected before OOM. But it seems to be not necessary so. Here is
what javadoc says

<quote>
 * <p> All soft references to softly-reachable objects are guaranteed to have
 * been cleared before the virtual machine throws an
 * <code>OutOfMemoryError</code>.  Otherwise no constraints are placed upon the
 * time at which a soft reference will be cleared or the order in which a set
 * of such references to different objects will be cleared.  Virtual machine
 * implementations are, however, encouraged to bias against clearing
 * recently-created or recently-used soft references.
</quote>

How do you read it?

On Mon, Jan 24, 2011 at 7:02 PM, Jochen Theodorou <[hidden email]> wrote:

> Am 24.01.2011 15:16, schrieb Alex Tkachman:
>>
>> I am under the same impression.
>>
>> The interesting fact I read from the same screenshot is that it is
>> proven now we generates tons of classes for the same method.
>
> as I said to Marc, that itself is not necessarily wrong. If 3000 places call
> ConfigSlurper#flatten, you get 3000 call sites that each has its own
> generated method to call ConfigSlurper#flatten.
>
>> Jochen, could you please have a look on CachedMethod and how we cache
>> constructors of generated classes. I start thinking that may be we
>> deal with classical concurrency issue on Java Memory Model.
>
> I am wondering in a different direction atm actually. The Constructor is a
> SoftReference. But what keeps the Constructor from being collected? If the
> constructor is not there, then we will have to recreate the method all the
> time.
>
> As for JMM... I doubt this is really a problem. The problem mentioned here
> appears after weeks of running the application. Even if each thread had its
> own variant, it would not lead to such problems. And a partially initiated
> SoftReference should cause other problems. A partially initiated Constructor
> too.
>
> bye blackdrag
>
> --
> Jochen "blackdrag" Theodorou
> The Groovy Project Tech Lead
> http://blackdragsview.blogspot.com/
> For Groovy programming sources visit http://groovy.codehaus.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>   http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

melix
Another idea. Marc, what are the VM options you are currently using ? In
particular, do you use the concurrent mark sweep GC ? If so, you might
need this option :
+XX:CMSClassUnloadingEnabled

This option allows permgen space recovery when using the CMS. It's not
enabled by default.

See http://www.md.pp.ru/~eu/jdk6options.html#CMSClassUnloadingEnabled 
<http://www.md.pp.ru/%7Eeu/jdk6options.html#CMSClassUnloadingEnabled>

Cédric

Le 24/01/2011 19:19, Alex Tkachman a écrit :

> As I mentioned before Constructor collecting is my first theory. I was
> under assumption that being on SoftReference means exactly that it is
> not collected before OOM. But it seems to be not necessary so. Here is
> what javadoc says
>
> <quote>
>   *<p>  All soft references to softly-reachable objects are guaranteed to have
>   * been cleared before the virtual machine throws an
>   *<code>OutOfMemoryError</code>.  Otherwise no constraints are placed upon the
>   * time at which a soft reference will be cleared or the order in which a set
>   * of such references to different objects will be cleared.  Virtual machine
>   * implementations are, however, encouraged to bias against clearing
>   * recently-created or recently-used soft references.
> </quote>
>
> How do you read it?
>
> On Mon, Jan 24, 2011 at 7:02 PM, Jochen Theodorou<[hidden email]>  wrote:
>> Am 24.01.2011 15:16, schrieb Alex Tkachman:
>>> I am under the same impression.
>>>
>>> The interesting fact I read from the same screenshot is that it is
>>> proven now we generates tons of classes for the same method.
>> as I said to Marc, that itself is not necessarily wrong. If 3000 places call
>> ConfigSlurper#flatten, you get 3000 call sites that each has its own
>> generated method to call ConfigSlurper#flatten.
>>
>>> Jochen, could you please have a look on CachedMethod and how we cache
>>> constructors of generated classes. I start thinking that may be we
>>> deal with classical concurrency issue on Java Memory Model.
>> I am wondering in a different direction atm actually. The Constructor is a
>> SoftReference. But what keeps the Constructor from being collected? If the
>> constructor is not there, then we will have to recreate the method all the
>> time.
>>
>> As for JMM... I doubt this is really a problem. The problem mentioned here
>> appears after weeks of running the application. Even if each thread had its
>> own variant, it would not lead to such problems. And a partially initiated
>> SoftReference should cause other problems. A partially initiated Constructor
>> too.
>>
>> bye blackdrag
>>
>> --
>> Jochen "blackdrag" Theodorou
>> The Groovy Project Tech Lead
>> http://blackdragsview.blogspot.com/
>> For Groovy programming sources visit http://groovy.codehaus.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>      http://xircles.codehaus.org/manage_email
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Marc Palmer Local
In reply to this post by Jochen Theodorou

On 24 Jan 2011, at 18:02, Jochen Theodorou wrote:

> Am 24.01.2011 15:16, schrieb Alex Tkachman:
>> I am under the same impression.
>>
>> The interesting fact I read from the same screenshot is that it is
>> proven now we generates tons of classes for the same method.
>
> as I said to Marc, that itself is not necessarily wrong. If 3000 places call ConfigSlurper#flatten, you get 3000 call sites that each has its own generated method to call ConfigSlurper#flatten.

OK, that is counter to my previous understanding.

However, I am pretty sure we don't have 2000 places that call my WcmContentRepositoryService.archiveStaleContent.

In fact, we have precisely... 1.

Marc


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Alex Tkachman
Marc,

Could you please try to apply attached patch to Groovy 1.7.x trunk and
use this version with your app. The idea of the patch is that instance
of generated call site keeps strong reference to Constructor, so it
will never be collected and so new call site class will never be
regenerated.

Alex

On Mon, Jan 24, 2011 at 8:21 PM, Marc Palmer <[hidden email]> wrote:

>
> On 24 Jan 2011, at 18:02, Jochen Theodorou wrote:
>
>> Am 24.01.2011 15:16, schrieb Alex Tkachman:
>>> I am under the same impression.
>>>
>>> The interesting fact I read from the same screenshot is that it is
>>> proven now we generates tons of classes for the same method.
>>
>> as I said to Marc, that itself is not necessarily wrong. If 3000 places call ConfigSlurper#flatten, you get 3000 call sites that each has its own generated method to call ConfigSlurper#flatten.
>
> OK, that is counter to my previous understanding.
>
> However, I am pretty sure we don't have 2000 places that call my WcmContentRepositoryService.archiveStaleContent.
>
> In fact, we have precisely... 1.
>
> Marc
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

Attempt_to_fix_GC_bug_of_Marc.patch (9K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Marc Palmer Local

On 24 Jan 2011, at 19:40, Alex Tkachman wrote:

> Marc,
>
> Could you please try to apply attached patch to Groovy 1.7.x trunk and
> use this version with your app. The idea of the patch is that instance
> of generated call site keeps strong reference to Constructor, so it
> will never be collected and so new call site class will never be
> regenerated.

Hi Alex,

I will try to find time to do this, but I have never built groovy from source, and don't even have a local source checkout.

Thanks
Marc


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Alex Tkachman
OK, I will build it for you

On Mon, Jan 24, 2011 at 8:49 PM, Marc Palmer <[hidden email]> wrote:

>
> On 24 Jan 2011, at 19:40, Alex Tkachman wrote:
>
>> Marc,
>>
>> Could you please try to apply attached patch to Groovy 1.7.x trunk and
>> use this version with your app. The idea of the patch is that instance
>> of generated call site keeps strong reference to Constructor, so it
>> will never be collected and so new call site class will never be
>> regenerated.
>
> Hi Alex,
>
> I will try to find time to do this, but I have never built groovy from source, and don't even have a local source checkout.
>
> Thanks
> Marc
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Trying to debug a permgen leak... what is this Class?

Marc Palmer Local

On 24 Jan 2011, at 19:51, Alex Tkachman wrote:

> OK, I will build it for you

OK, that would be cool. I can probably drop the JAR into the existing WAR folder on the server and restart, and start grabbing periodic heapdumps and see how it progresses.

MArc


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


1234
Loading...