About the Phoenix plan for Groovy

classic Classic list List threaded Threaded
13 messages Options
12
Reply | Threaded
Open this post in threaded view
|

About the Phoenix plan for Groovy

Daniel Sun
Hi all,

      I have a plan named "Phoenix" to share. As we all know, STC is buggy,
one of reasons is that STC lacks tests during development. So I make a plan
to let STC mature enough:

1) Make the Parrot parser be able to parse pure Java source to Groovy AST;
2) Use STC to analyze the AST and generate the bytecode;
3) Run the existing tests of some famous Java projects, e.g. Spring,
Hibernate, Guava, Hadoop, etc. ;
4) Check the test result. If all tests pass, our Phoenix plan is completed.

    Another benefit of the plan is that we will have a brand new joint
compiler, no stub generation is required :-)

    Any thoughts?

Cheers,
Daniel.Sun



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Reply | Threaded
Open this post in threaded view
|

Re: About the Phoenix plan for Groovy

Daniel Sun
Groovydoc is easy to get via Parrot now.
When step 1 of Phoenix plan is completed, Javadoc will be  easy to get too
;-)

Cheers,
Daniel.Sun




--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Reply | Threaded
Open this post in threaded view
|

Re: About the Phoenix plan for Groovy

Jochen Theodorou
In reply to this post by Daniel Sun
On 08.04.2018 03:58, Daniel Sun wrote:

> Hi all,
>
>        I have a plan named "Phoenix" to share. As we all know, STC is buggy,
> one of reasons is that STC lacks tests during development. So I make a plan
> to let STC mature enough:
>
> 1) Make the Parrot parser be able to parse pure Java source to Groovy AST;
> 2) Use STC to analyze the AST and generate the bytecode;
> 3) Run the existing tests of some famous Java projects, e.g. Spring,
> Hibernate, Guava, Hadoop, etc. ;
> 4) Check the test result. If all tests pass, our Phoenix plan is completed.
>
>      Another benefit of the plan is that we will have a brand new joint
> compiler, no stub generation is required :-)
>
>      Any thoughts?

basically a good idea.

Though flow typing will cause the bytecode to differ. Very simple example

class X {
   private int y
   int foo(){return 1}
   boolean test(Z z) {
     X x = z;
     return y == x.y && foo() == x.foo()
   }
}
class Z extends X {
   int foo(){return 2}
}

In Java you can do this, in static Groovy the compiler may assign the
type Z to x, thus y is not accessible.... Can be fixed of course. Next
is that the compiler may generate a call to Z#foo instead of X#foo. And
while in this example it arguable could be changed, once Z defines a
more specific method, than X, that method is to be taken.

To me this means: to be able to compile like Java, you have to be able
to turn flow typing off.

And then there is the question of when things are equal. You really want
to go with bit-by-bit equality? What about the "implement GroovyObject"
for example? And then we do a lot of storing results in local variables,
that then are not used. And a lot of other quirks here and there.

And the Java compiler changes too. For example in

Thread.run(
   () -> {
     println 1
   }
)

the Java compiler does not create line number information for the second
line, which already caused my some puzzling moment when debugging code.
Recently there was a suggestion to change this. Just to illustrate that
the bytecode will depend on the Java compiler version and there will be
changes even in minor versions.

If you do not go by an automated version, you can obviously not just
feed it a huge amount of Java code and compare. Reducing the bytecode
test to the method calls and variable types may already do the trick,
but the bytecode does not contain that much generics information itself.

I would include the OpenJDK in there, especially the tests for generics.
And especially the tests for things that are supposed to *not* compile.
Of course in those cases you cannot just compare the bytecode (as there
is none)

Finally, not wanting to sound negative, but finding bugs is one thing,
fixing them is another and we do not have a shortage on reported bugs
for the static compiler

bye Jochen
Reply | Threaded
Open this post in threaded view
|

Re: About the Phoenix plan for Groovy

Daniel Sun
Hi Jochen,

>      To me this means: to be able to compile like Java, you have to be
> able
> to turn flow typing off.
      Thanks for your reminding :-)

>  You really want
> to go with bit-by-bit equality?
       I do not care about how to generating same bytecode as Java's but I
want the same execution result. That's like the relationship between IBM's
J9 and Oracle's Hotspot. it is the backend of the compiler
       As for the frontend, Groovy syntax is much complex than Java's, so I
plan to make Parrot parser support Java syntax as a dialect.

> I would include the OpenJDK in there, especially the tests for generics.
> And especially the tests for things that are supposed to *not* compile.
       OK. I'll take into account  it :-)

> Finally, not wanting to sound negative, but finding bugs is one thing,
> fixing them is another and we do not have a shortage on reported bugs
> for the static compiler
       We always fix bugs but do not know when we will complete. The Phoenix
plan can make the target much more clear. The development process is similar
to Parrot parser, which parses the source code of grails, gradle, spock, geb
and groovy itself for test compatibility, so the quality of the Parrot
parser is assured ;-)


Cheers,
Daniel.Sun







--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
MG
Reply | Threaded
Open this post in threaded view
|

Re: About the Phoenix plan for Groovy

MG
It seems Daniel has a point here: Knowing the whole landscape of bugs in the static compiler cannot be a bad thing in itself, right ? And it might prohibit fixing one bug reported by a Groovy user through a quick fix, when it is actually part of a bigger problem.

I would however keep bugs found this way somehow distinguishable in Jira from ones reported by users, since there will probably be some exotic/obscure edge case bugs which no one really notices in practice, and which won 't be fixed for some time.

The good news: In Groovy you have fallback to dynamic compilation, if static for some reason fails, so unless the bug is in a really time critical part of the code, you always* have a workable workaround at hand...

@Cents(10)
mg

*except in the Minecraft Forge obfuscation case, of course ;;;-)



-------- Ursprüngliche Nachricht --------
Von: Daniel Sun <[hidden email]>
Datum: 08.04.18 13:18 (GMT+01:00)
Betreff: Re: About the Phoenix plan for Groovy

Hi Jochen,

>      To me this means: to be able to compile like Java, you have to be
> able
> to turn flow typing off.
      Thanks for your reminding :-)

>  You really want
> to go with bit-by-bit equality?
       I do not care about how to generating same bytecode as Java's but I
want the same execution result. That's like the relationship between IBM's
J9 and Oracle's Hotspot. it is the backend of the compiler
       As for the frontend, Groovy syntax is much complex than Java's, so I
plan to make Parrot parser support Java syntax as a dialect.

> I would include the OpenJDK in there, especially the tests for generics.
> And especially the tests for things that are supposed to *not* compile.
       OK. I'll take into account  it :-)

> Finally, not wanting to sound negative, but finding bugs is one thing,
> fixing them is another and we do not have a shortage on reported bugs
> for the static compiler
       We always fix bugs but do not know when we will complete. The Phoenix
plan can make the target much more clear. The development process is similar
to Parrot parser, which parses the source code of grails, gradle, spock, geb
and groovy itself for test compatibility, so the quality of the Parrot
parser is assured ;-)


Cheers,
Daniel.Sun







--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
MG
Reply | Threaded
Open this post in threaded view
|

Re: About the Phoenix plan for Groovy

MG
In reply to this post by Daniel Sun
It seems Daniel has a point here: Knowing the whole landscape of bugs in the static compiler cannot be a bad thing in itself, right ? And it might prohibit fixing one bug reported by a Groovy user through a quick fix, when it is actually part of a bigger problem.

I would however keep bugs found this way somehow distinguishable in Jira from ones reported by users, since there will probably be some exotic/obscure edge case bugs which no one really notices in practice, and which won 't be fixed for some time.

The good news: In Groovy you have fallback to dynamic compilation, if static for some reason fails, so unless the bug is in a really time critical part of the code, you always* have a workable workaround at hand...

@Cents(10)
mg

*except in the Minecraft Forge obfuscation case, of course ;;;-)



-------- Ursprüngliche Nachricht --------
Von: Daniel Sun <[hidden email]>
Datum: 08.04.18 13:18 (GMT+01:00)
Betreff: Re: About the Phoenix plan for Groovy

Hi Jochen,

>      To me this means: to be able to compile like Java, you have to be
> able
> to turn flow typing off.
      Thanks for your reminding :-)

>  You really want
> to go with bit-by-bit equality?
       I do not care about how to generating same bytecode as Java's but I
want the same execution result. That's like the relationship between IBM's
J9 and Oracle's Hotspot. it is the backend of the compiler
       As for the frontend, Groovy syntax is much complex than Java's, so I
plan to make Parrot parser support Java syntax as a dialect.

> I would include the OpenJDK in there, especially the tests for generics.
> And especially the tests for things that are supposed to *not* compile.
       OK. I'll take into account  it :-)

> Finally, not wanting to sound negative, but finding bugs is one thing,
> fixing them is another and we do not have a shortage on reported bugs
> for the static compiler
       We always fix bugs but do not know when we will complete. The Phoenix
plan can make the target much more clear. The development process is similar
to Parrot parser, which parses the source code of grails, gradle, spock, geb
and groovy itself for test compatibility, so the quality of the Parrot
parser is assured ;-)


Cheers,
Daniel.Sun







--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Reply | Threaded
Open this post in threaded view
|

Re: About the Phoenix plan for Groovy

Daniel.Sun
Hi mg,

     I just do not like the feeling that we walk in an endless road ;-)
     The Phoenix plan will be a long road, but we can see the end of it.

      I will enhance the Parrot parser to support Java grammar as a dialect
at first, a journey of a thousand miles begins with single step :-)

Cheers,
Daniel.Sun



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Daniel Sun
Apache Groovy committer

Blog: http://blog.sunlan.me
Twitter: @daniel_sun
Reply | Threaded
Open this post in threaded view
|

Re: About the Phoenix plan for Groovy

Cédric Champeau
In reply to this post by Daniel Sun


2018-04-08 3:58 GMT+02:00 Daniel Sun <[hidden email]>:
Hi all,

      I have a plan named "Phoenix" to share. As we all know, STC is buggy,
one of reasons is that STC lacks tests during development.

It's interesting you put test coverage as one reason that the STC is buggy, but the only reason you mention. As the author of most of the code here, I'd argue that there are way more STC tests than any other feature in the language :) Almost 1/3 of the test suite is about STC/static compilation. I think the reasons it's buggy are not to be found here, but on the complexity of building a type safe feature on top of an AST that was primarily designed years ago for a dynamic language. In particular, the ClassNode infrastructure, and the way it handles generics, was a true nightmare. Second, type inference is extremely hard. Flow typing is also something complex to get right, in particular discovering variable capture, or "effectively final" variables. The last part, bytecode generation, is very easy in comparison. Eventually, the STC was designed as an AST visitor, which, unfortunately, doesn't allow to carry much context (the various visit methods do not take a context argument, which would have helped a lot).

So I'm all in for a new version, but that's not a simple task. Especially when people disagree on what types should be inferred where. Look at what happens with Java and the semantics of "var", or list literals, and lambda type inference. This is the tricky part, which _always_ introduces bugs. The Java compiler itself is not bug free...
 
So I make a plan
to let STC mature enough:

1) Make the Parrot parser be able to parse pure Java source to Groovy AST;
2) Use STC to analyze the AST and generate the bytecode;
3) Run the existing tests of some famous Java projects, e.g. Spring,
Hibernate, Guava, Hadoop, etc. ;
4) Check the test result. If all tests pass, our Phoenix plan is completed.

    Another benefit of the plan is that we will have a brand new joint
compiler, no stub generation is required :-)

    Any thoughts?

Cheers,
Daniel.Sun



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Reply | Threaded
Open this post in threaded view
|

Re: About the Phoenix plan for Groovy

Daniel.Sun
Hi Cédric,

       At first we all admit your effort on STC is great :-)  

       I usually try to use STC for better performance, but sadly I have to
turn to dynamic mode sometimes. Recently I wrote a hadoop example in Java,
then copied the Java code as Groovy code and tried to compile with STC,
failed to compile because of generics issue. So I believe STC had been
tested for many scenario as you said, but it had not been tested with the
code in some real projects, e.g. Spring, Hibernate, etc. The Phoenix plan
tries to find a way to polish the STC and not to re-invent the wheels.

       Finally we wish you could set aside some time to fix some existing
bugs, some of which are hard for us to fix but may be not that hard for you
- the creator of STC ;-)

Cheers,
Daniel.Sun




--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Daniel Sun
Apache Groovy committer

Blog: http://blog.sunlan.me
Twitter: @daniel_sun
Reply | Threaded
Open this post in threaded view
|

Re: About the Phoenix plan for Groovy

Cédric Champeau


2018-04-08 19:31 GMT+02:00 Daniel.Sun <[hidden email]>:
Hi Cédric,

       At first we all admit your effort on STC is great :-)

       I usually try to use STC for better performance, but sadly I have to
turn to dynamic mode sometimes. Recently I wrote a hadoop example in Java,
then copied the Java code as Groovy code and tried to compile with STC,
failed to compile because of generics issue. So I believe STC had been
tested for many scenario as you said, but it had not been tested with the
code in some real projects, e.g. Spring, Hibernate, etc. The Phoenix plan
tries to find a way to polish the STC and not to re-invent the wheels.

It was tested on real code too, I can assure you :)

       Finally we wish you could set aside some time to fix some existing
bugs, some of which are hard for us to fix but may be not that hard for you
- the creator of STC ;-)

Honestly I don't have much time to work on Groovy. And if I had, the generics STC bugs would be the last thing I'd like to get my hands into. The ClassNode infrastructure was retrofitted to introduce generics back in 1.5, but since it was for dynamic world, it wasn't really designed for what we do now. So, every bugfix you make is likely to introduce another bug somewhere else. That makes the process very painful. More so when type inference enters the game. Jochen and I spent a lot of time writing, rewriting helper methods to make this easier, but it's just too hard for the little time I can afford.

off topic: why is incubator always in CC of the emails you write? Looks like your email client is misconfigured somehow.

12