Making DGM extensible / Extensible MetaClass

classic Classic list List threaded Threaded
12 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Making DGM extensible / Extensible MetaClass

graemer
Hi Devs,

At the moment it seems almost impossible to extends DGM without doing
some surgery. So I want to put forward some proposals now to get
around this. The first proposal is that we need some way
extending/replacing the MetaClassRegistry

As a minimum the methods lookup and registerMethods need to be made
protected and Invoker changed to allow a customer registry to bet set.
This will allow people to provide their own extended DGM. This is the
simpler of the two proposals.

My second proposal is a little more radical and it is what I believe
Groovy needs to do to be able to compete with Ruby and the like long
term. Essentially MetaClass needs to implement GroovyObject and needs
to be modifiable at runtime.

You need to able to do this:

def myString = "blah"

myString.metaClass.threeTimes = {
      it * 3
}

myString.threeTimes()

This should change the MetaClass in the registry at the class level
and effect all new instances of said class. Obviously there a
threading issues here so a developer must make sure this is done on
application load. This should be enabled for properites too:

// property
myString.metaClass.property = "blah"

It should also be possible with static methods, ideally you could do
someething like this:

myString.class.threeTimes = {  ...  }

or

String.threeTimes = { ... }

Although I'm not sure how this would be possible. Maybe a MetaClass
for java.lang.Class?

This will make Groovy a truely dyamic language with the ability to
change classes at runtime. Without this feature and with the
relatively large amount of work it is at the moment to implement a
custom MetaClass Groovy will suffer long term.

Grails is going to need one of the two proposals. In the short terms I
would settle for proposal 1, but in the long term I feel Groovy will
benefit most from proposal 2.

Thoughts?

--
Graeme Rocher
Grails Project Lead
http://grails.org

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: Making DGM extensible / Extensible MetaClass

tugwilson

On 26 Sep 2006, at 12:38, Graeme Rocher wrote:

> Hi Devs,
>
> At the moment it seems almost impossible to extends DGM without doing
> some surgery. So I want to put forward some proposals now to get
> around this. The first proposal is that we need some way
> extending/replacing the MetaClassRegistry
>
> As a minimum the methods lookup and registerMethods need to be made
> protected and Invoker changed to allow a customer registry to bet set.
> This will allow people to provide their own extended DGM. This is the
> simpler of the two proposals.
>
> My second proposal is a little more radical and it is what I believe
> Groovy needs to do to be able to compete with Ruby and the like long
> term. Essentially MetaClass needs to implement GroovyObject and needs
> to be modifiable at runtime.
>
> You need to able to do this:
>
> def myString = "blah"
>
> myString.metaClass.threeTimes = {
>      it * 3
> }
>
> myString.threeTimes()
>
> This should change the MetaClass in the registry at the class level
> and effect all new instances of said class. Obviously there a
> threading issues here so a developer must make sure this is done on
> application load. This should be enabled for properites too:
>
> // property
> myString.metaClass.property = "blah"
>
> It should also be possible with static methods, ideally you could do
> someething like this:
>
> myString.class.threeTimes = {  ...  }
>
> or
>
> String.threeTimes = { ... }
>
> Although I'm not sure how this would be possible. Maybe a MetaClass
> for java.lang.Class?
>
> This will make Groovy a truely dyamic language with the ability to
> change classes at runtime. Without this feature and with the
> relatively large amount of work it is at the moment to implement a
> custom MetaClass Groovy will suffer long term.
>
> Grails is going to need one of the two proposals. In the short terms I
> would settle for proposal 1, but in the long term I feel Groovy will
> benefit most from proposal 2.
>
> Thoughts?

I'd like to clarify what you are trying to achieve before commenting  
on your suggested change..

As I read this you would like a way of adding arbitrary methods to  
arbitrary classes at JVM startup time. (actually not quite at startup  
time but very early on in the lifecycle of the application).

You do *not* want to add arbitrary methods to arbitrary classes at  
arbitrary times in the program's execution.

Is that right?

Also can you clarify what you mean by adding properties? If I can ass  
methods I can ass get/set methods and therefor ass a property. That's  
trivial. However do you want also to add a field on the existing  
class to hold a value? That's non trivial and may not be practically  
possible for non GroovyObjects (and quite costly for GroovyObjects).

John Wilson
The Wilson Partnership
web http://www.wilson.co.uk
blog http://eek.ook.org



---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

[groovy-dev] Re: Making DGM extensible / Extensible MetaClass

graemer
On 9/26/06, John Wilson <[hidden email]> wrote:

>
> On 26 Sep 2006, at 12:38, Graeme Rocher wrote:
>
> > Hi Devs,
> >
> > At the moment it seems almost impossible to extends DGM without doing
> > some surgery. So I want to put forward some proposals now to get
> > around this. The first proposal is that we need some way
> > extending/replacing the MetaClassRegistry
> >
> > As a minimum the methods lookup and registerMethods need to be made
> > protected and Invoker changed to allow a customer registry to bet set.
> > This will allow people to provide their own extended DGM. This is the
> > simpler of the two proposals.
> >
> > My second proposal is a little more radical and it is what I believe
> > Groovy needs to do to be able to compete with Ruby and the like long
> > term. Essentially MetaClass needs to implement GroovyObject and needs
> > to be modifiable at runtime.
> >
> > You need to able to do this:
> >
> > def myString = "blah"
> >
> > myString.metaClass.threeTimes = {
> >      it * 3
> > }
> >
> > myString.threeTimes()
> >
> > This should change the MetaClass in the registry at the class level
> > and effect all new instances of said class. Obviously there a
> > threading issues here so a developer must make sure this is done on
> > application load. This should be enabled for properites too:
> >
> > // property
> > myString.metaClass.property = "blah"
> >
> > It should also be possible with static methods, ideally you could do
> > someething like this:
> >
> > myString.class.threeTimes = {  ...  }
> >
> > or
> >
> > String.threeTimes = { ... }
> >
> > Although I'm not sure how this would be possible. Maybe a MetaClass
> > for java.lang.Class?
> >
> > This will make Groovy a truely dyamic language with the ability to
> > change classes at runtime. Without this feature and with the
> > relatively large amount of work it is at the moment to implement a
> > custom MetaClass Groovy will suffer long term.
> >
> > Grails is going to need one of the two proposals. In the short terms I
> > would settle for proposal 1, but in the long term I feel Groovy will
> > benefit most from proposal 2.
> >
> > Thoughts?
>
> I'd like to clarify what you are trying to achieve before commenting
> on your suggested change..

Ok :-)

>
> As I read this you would like a way of adding arbitrary methods to
> arbitrary classes at JVM startup time. (actually not quite at startup
> time but very early on in the lifecycle of the application).
>
> You do *not* want to add arbitrary methods to arbitrary classes at
> arbitrary times in the program's execution.
Not necessarily, the programmer could add the method later, but of
course they would have to be careful in doing so to avoid threading
issues. Anyway they would be the same issues you encounter in other
languages like Ruby and even JavaScript with its prototype objects. If
you don't register these types of methods in a timely manner you will
run into problems.

But it is the ability to do this that is key. Having the mechanism
that is easy and accessible and doesn't require advanced knowledge of
the workins of MetaClass as it does now. Its the accessibility of the
feature. The current system is extremely error prone as you can easily
introduce all sorts of problems when implementing custom meta classes.

Much of the proposal is taking the burden of this off the developers
shoulders. I would be quite happy to implement this feature myself
actually if I was allowed to, but obviously it is dependant on the
MetaClass work that is currently under way.

>
> Is that right?
>
> Also can you clarify what you mean by adding properties? If I can ass
> methods I can ass get/set methods and therefor ass a property. That's
> trivial. However do you want also to add a field on the existing
> class to hold a value? That's non trivial and may not be practically
> possible for non GroovyObjects (and quite costly for GroovyObjects).
Properties are sufficient in my view. Fields are rarely necessary.

>
> John Wilson
> The Wilson Partnership
> web http://www.wilson.co.uk
> blog http://eek.ook.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>


--
Graeme Rocher
Grails Project Lead
http://grails.org

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: Making DGM extensible / Extensible MetaClass

tugwilson

On 26 Sep 2006, at 13:01, Graeme Rocher wrote:

> Not necessarily, the programmer could add the method later, but of
> course they would have to be careful in doing so to avoid threading
> issues. Anyway they would be the same issues you encounter in other
> languages like Ruby and even JavaScript with its prototype objects. If
> you don't register these types of methods in a timely manner you will
> run into problems.
>
> But it is the ability to do this that is key. Having the mechanism
> that is easy and accessible and doesn't require advanced knowledge of
> the workins of MetaClass as it does now. Its the accessibility of the
> feature. The current system is extremely error prone as you can easily
> introduce all sorts of problems when implementing custom meta classes.
>
> Much of the proposal is taking the burden of this off the developers
> shoulders. I would be quite happy to implement this feature myself
> actually if I was allowed to, but obviously it is dependant on the
> MetaClass work that is currently under way.


Ok lets see what we can do....

Firstly you need to be aware that dynamically adding a method to a  
class via the MetaClass will in future be a *very* expensive  
operation. I have no doubt that future MetaClass implementations will  
dynamically compile the method selection and dispatch logic to get  
the best performance possible. If you add another permanent method  
then the old dispatch code needs to be discarded and a new version  
compiled - then you have the JIT warm up period to go through before  
you get back to the old speed. So if you are adding several methods  
you really want to add them all at once. In my view this is best done  
using Category styles classes with static methods.

[The obvious question following on form the above is "how will  
Categories be implemented when the dispatch code is all compiled?".  
Good Question!  I think that we will do interpreted dispatch for  
those threads executing inside a Category and compiled dispatch for  
those threads not inside a Category. However we will need to mark a  
MetaClass as being touched by a Category and then check if the thread  
is in a Category when the MetaClass is accessed. This will impact all  
threads using this MetaClass, I'm afraid]

So I would suggest that we have a mechanism of specifying what  
"universal Categories" should be applied to the MetaClasses at start  
up time. If this is acceptable we can go on to discuss how this could  
be done.

Now we are left with your requirement to add method(s) to arbitrary  
classes at arbitrary times in the program's execution. I'm working at  
a disadvantage here because I can't think of a use case:)

I *really* think it's not acceptable for one thread to make a change  
which immediately changes the behaviour of all the other threads in a  
JVM.

I wonder if we could do some ClassLoader macgic such we could  
instantiate a ClassLoader with a set of universal Catagories so that  
all classes loaded by the new ClassLoader world see the new behaviour.

Do you think that is possible and if so would it do what you want?


John Wilson
The Wilson Partnership
web http://www.wilson.co.uk
blog http://eek.ook.org



---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: Making DGM extensible / Extensible MetaClass

graemer
On 9/26/06, John Wilson <[hidden email]> wrote:

>
> On 26 Sep 2006, at 13:01, Graeme Rocher wrote:
>
> > Not necessarily, the programmer could add the method later, but of
> > course they would have to be careful in doing so to avoid threading
> > issues. Anyway they would be the same issues you encounter in other
> > languages like Ruby and even JavaScript with its prototype objects. If
> > you don't register these types of methods in a timely manner you will
> > run into problems.
> >
> > But it is the ability to do this that is key. Having the mechanism
> > that is easy and accessible and doesn't require advanced knowledge of
> > the workins of MetaClass as it does now. Its the accessibility of the
> > feature. The current system is extremely error prone as you can easily
> > introduce all sorts of problems when implementing custom meta classes.
> >
> > Much of the proposal is taking the burden of this off the developers
> > shoulders. I would be quite happy to implement this feature myself
> > actually if I was allowed to, but obviously it is dependant on the
> > MetaClass work that is currently under way.
>
>
> Ok lets see what we can do....
>
> Firstly you need to be aware that dynamically adding a method to a
> class via the MetaClass will in future be a *very* expensive
> operation. I have no doubt that future MetaClass implementations will
> dynamically compile the method selection and dispatch logic to get
> the best performance possible. If you add another permanent method
> then the old dispatch code needs to be discarded and a new version
> compiled - then you have the JIT warm up period to go through before
> you get back to the old speed. So if you are adding several methods
> you really want to add them all at once. In my view this is best done
> using Category styles classes with static methods.


I'm happy to compromise here, but I think the point is making the
point of entry lower for implementing functionality like this. If this
is a category then so be it. I think that adding the ability to add
methods dynamically at runtime is interesting, but maybe less useful
the categories.

I would say something like this would be useful:

class MyCategory {
        static String duplicate(String self) {
                  return self*2
        }
}

GroovyClassLoader gcl = new GroovyClassLoader()
gcl.addCategory(MyCategory.class)

At least then you've lowered the barrier significantly. You have
provided a way for people to register "universal" categories without
having to worry about implementing MetaClass and causing horrific
problems with scoping/method resolution/return values going wrong when
there are bugs.

Although personally I don't think it is as slick as the Ruby/SmallTalk
solution which is more like what I mentioned in proposal 1 at least it
is something.

Graeme

>
> [The obvious question following on form the above is "how will
> Categories be implemented when the dispatch code is all compiled?".
> Good Question!  I think that we will do interpreted dispatch for
> those threads executing inside a Category and compiled dispatch for
> those threads not inside a Category. However we will need to mark a
> MetaClass as being touched by a Category and then check if the thread
> is in a Category when the MetaClass is accessed. This will impact all
> threads using this MetaClass, I'm afraid]
>
> So I would suggest that we have a mechanism of specifying what
> "universal Categories" should be applied to the MetaClasses at start
> up time. If this is acceptable we can go on to discuss how this could
> be done.
>
> Now we are left with your requirement to add method(s) to arbitrary
> classes at arbitrary times in the program's execution. I'm working at
> a disadvantage here because I can't think of a use case:)
>
> I *really* think it's not acceptable for one thread to make a change
> which immediately changes the behaviour of all the other threads in a
> JVM.
>
> I wonder if we could do some ClassLoader macgic such we could
> instantiate a ClassLoader with a set of universal Catagories so that
> all classes loaded by the new ClassLoader world see the new behaviour.
>
> Do you think that is possible and if so would it do what you want?
>
>
> John Wilson
> The Wilson Partnership
> web http://www.wilson.co.uk
> blog http://eek.ook.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>


--
Graeme Rocher
Grails Project Lead
http://grails.org

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

[groovy-dev] Re: Making DGM extensible / Extensible MetaClass

tugwilson

On 26 Sep 2006, at 15:00, Graeme Rocher wrote:

> I would say something like this would be useful:
>
> class MyCategory {
>        static String duplicate(String self) {
>                  return self*2
>        }
> }
>
> GroovyClassLoader gcl = new GroovyClassLoader()
> gcl.addCategory(MyCategory.class)


That's not quite what I suggested:)

GroovyClassLoader gcl = new GroovyClassLoader(MyCategory.class)

is what I had in mind.

Lets see what others thing about this. I'm particularly interested in  
Blackdrag's view in this - especially as to if this is doable.


John Wilson
The Wilson Partnership
web http://www.wilson.co.uk
blog http://eek.ook.org



---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

[groovy-dev] Re: Making DGM extensible / Extensible MetaClass

graemer
On 9/26/06, John Wilson <[hidden email]> wrote:

>
> On 26 Sep 2006, at 15:00, Graeme Rocher wrote:
>
> > I would say something like this would be useful:
> >
> > class MyCategory {
> >        static String duplicate(String self) {
> >                  return self*2
> >        }
> > }
> >
> > GroovyClassLoader gcl = new GroovyClassLoader()
> > gcl.addCategory(MyCategory.class)
>
>
> That's not quite what I suggested:)
>
> GroovyClassLoader gcl = new GroovyClassLoader(MyCategory.class)
>
> is what I had in mind.

Yes but that implies that you can have only one category class, as a
minimum you should be abl to do

Class[] categories = ... // create a list of them
gcl = new GroovyClassLoader(categories)

>
> Lets see what others thing about this. I'm particularly interested in
> Blackdrag's view in this - especially as to if this is doable.

Indeed.

Graeme

>
>
> John Wilson
> The Wilson Partnership
> web http://www.wilson.co.uk
> blog http://eek.ook.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>


--
Graeme Rocher
Grails Project Lead
http://grails.org

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: Making DGM extensible / Extensible MetaClass

Jochen Theodorou
I think there are some small points I currently do not understand.

First. We are talking about optimization to the method selection. What
is the matter of redoing that? We haven't have that now, I maybe we will
be able to use parts of it without bytecode production. And even then..
we can recompile. What is the problem? sure, it is costly, but normally
such things are not done in runtime critical parts. Maybe it is possible
to chain the method selections...

The alternative is setting a custome MetaClass... but doesn't that
approach have the same problems? And isn't it the same vor Categories?
Regardless the thread visibility, we do add methods.

btw, I can exchange the MetaClass for java classes by using

InvokerHelper.getinstance().getMetaRegistry().
setMetaClass(klass, metaClass)

Then the next thing... adding methods.. it is not as simple is in
Smalltalk, since you can't overload methods in smalltalk. Adding a
method my result in an problem when doing method dispatching... for
exmaple if I have

foo(String,Object)
foo(Object,String)

and a call foo(String,String), then none of the methods above is better,
the runtime is unable to select a concrete method. So adding the secound
method must throw an excpetion. A problem is also overwriting existing
methods, we must not allow that unless the user really wants to do that.

So I don't think it is something we can't allow, not in terms of
semantics, and not in terms of performance. Instead I suggest to wirte a
page covering performance pitfalls and explaining what they shouldn't do.

bye blackdrag

--
Jochen Theodorou
Groovy Tech Lead
http://blackdragsview.blogspot.com/

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: Making DGM extensible / Extensible MetaClass

graemer
On 9/26/06, Jochen Theodorou <[hidden email]> wrote:

> I think there are some small points I currently do not understand.
>
> First. We are talking about optimization to the method selection. What
> is the matter of redoing that? We haven't have that now, I maybe we will
> be able to use parts of it without bytecode production. And even then..
> we can recompile. What is the problem? sure, it is costly, but normally
> such things are not done in runtime critical parts. Maybe it is possible
> to chain the method selections...
>
> The alternative is setting a custome MetaClass... but doesn't that
> approach have the same problems? And isn't it the same vor Categories?
> Regardless the thread visibility, we do add methods.

So you're suggesting my first proposal is a possibility?

>
> btw, I can exchange the MetaClass for java classes by using
>
> InvokerHelper.getinstance().getMetaRegistry().
> setMetaClass(klass, metaClass)
>
> Then the next thing... adding methods.. it is not as simple is in
> Smalltalk, since you can't overload methods in smalltalk. Adding a
> method my result in an problem when doing method dispatching... for
> exmaple if I have
>
> foo(String,Object)
> foo(Object,String)
>
> and a call foo(String,String), then none of the methods above is better,
> the runtime is unable to select a concrete method. So adding the secound
> method must throw an excpetion. A problem is also overwriting existing
> methods, we must not allow that unless the user really wants to do that.

Indeed, but that is part of the problem, at the moment with custom
MetaClasses as a developer you have to think about these things. You
have to think "what happens if the method exists?" or "what if its
overloaded?".

This only further increases the point on entry. If, however, there was
something built into Groovy that would throw appropriate errors with
good errror messages if you tried to replace an existing method or
causes overloading issues like your example above I think thats
perfectly acceptable and in fact a good thing.

>
> So I don't think it is something we can't allow, not in terms of
> semantics, and not in terms of performance. Instead I suggest to wirte a
> page covering performance pitfalls and explaining what they shouldn't do.
We can start a Wiki page about the proposal if necessary, but lets see
if an agreement can be reached about the issues first

Graeme

>
> bye blackdrag
>
> --
> Jochen Theodorou
> Groovy Tech Lead
> http://blackdragsview.blogspot.com/
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>


--
Graeme Rocher
Grails Project Lead
http://grails.org

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: Making DGM extensible / Extensible MetaClass

Jochen Theodorou
sorry, posted only to the jsr list

Graeme Rocher schrieb:

> On 9/26/06, Jochen Theodorou <[hidden email]> wrote:
>> I think there are some small points I currently do not understand.
>>
>> First. We are talking about optimization to the method selection. What
>> is the matter of redoing that? We haven't have that now, I maybe we will
>> be able to use parts of it without bytecode production. And even then..
>> we can recompile. What is the problem? sure, it is costly, but normally
>> such things are not done in runtime critical parts. Maybe it is possible
>> to chain the method selections...
>>
>> The alternative is setting a custome MetaClass... but doesn't that
>> approach have the same problems? And isn't it the same vor Categories?
>> Regardless the thread visibility, we do add methods.
>
> So you're suggesting my first proposal is a possibility?

we could have multithreaded ategories... The approach with closures is..

object.metaClass.addMethod("foo") { println "hi" }

we could something like that for closures. This actually adds 2 methods,
foo() and foo(Object). In fact I used that already in a modified version
some time ago.

[...]

> at the moment with custom
> MetaClasses as a developer you have to think about these things. You
> have to think "what happens if the method exists?" or "what if its
> overloaded?".
>
> This only further increases the point on entry. If, however, there was
> something built into Groovy that would throw appropriate errors with
> good errror messages if you tried to replace an existing method or
> causes overloading issues like your example above I think thats
> perfectly acceptable and in fact a good thing.

yes, that is what I want to do. It is not much of a problem to find the
problematic cases I think.

ah.. but one moire thing... why the hell new
GroovyClassLoader(Foo.class) ???

bye blackdrag

--
Jochen Theodorou
Groovy Tech Lead
http://blackdragsview.blogspot.com/

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

12