[PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance

classic Classic list List threaded Threaded
17 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance

Daniel Sun
Hi all,

[Background]
       Groovy's joint compiler will generate a lot of stubs for groovy
source files, many of the stubs are useless and written to disk and clean
later. When project contains a lot of groovy source files, the performance
of compiling is not good.

[Proposal]
      I propose to add an option(e.g. `groovy.generate.stubs.in.memory`) to
generate stubs in in-momery file system[1]. We can get the generated stub
files from the the in-memory file system with `StandardJavaFileManager`[2] .
Here is the `JavaCompiler` usage[3].

[Benefits]
     We can avoid writing lots of stub files to disk and subsequent
cleaning, in addition, reading stub files from memory will be much faster
than reading from disk.


      Any thoughts?

Cheers,
Daniel.Sun
[1] https://github.com/google/jimfs
[2]
https://docs.oracle.com/javase/7/docs/api/javax/tools/StandardJavaFileManager.html
[3]
https://github.com/danielsun1106/SmartASMifier/blob/master/SmartASMifier.groovy#L81-L88




--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance

Cédric Champeau
It's an interesting idea. I like that it's hidden behind a flag, because sometimes you want to keep the stubs. That must be the case for IDEs that implement "some kind" of incremental compilation.

Le ven. 18 janv. 2019 à 09:44, Daniel Sun <[hidden email]> a écrit :
Hi all,

[Background]
       Groovy's joint compiler will generate a lot of stubs for groovy
source files, many of the stubs are useless and written to disk and clean
later. When project contains a lot of groovy source files, the performance
of compiling is not good.

[Proposal]
      I propose to add an option(e.g. `groovy.generate.stubs.in.memory`) to
generate stubs in in-momery file system[1]. We can get the generated stub
files from the the in-memory file system with `StandardJavaFileManager`[2] .
Here is the `JavaCompiler` usage[3].

[Benefits]
     We can avoid writing lots of stub files to disk and subsequent
cleaning, in addition, reading stub files from memory will be much faster
than reading from disk.


      Any thoughts?

Cheers,
Daniel.Sun
[1] https://github.com/google/jimfs
[2]
https://docs.oracle.com/javase/7/docs/api/javax/tools/StandardJavaFileManager.html
[3]
https://github.com/danielsun1106/SmartASMifier/blob/master/SmartASMifier.groovy#L81-L88




--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance

Paolo Di Tommaso
In reply to this post by Daniel Sun
This would be indeed very useful. The current join compilation tend to be time consuming. 

p

On Fri, Jan 18, 2019 at 9:44 AM Daniel Sun <[hidden email]> wrote:
Hi all,

[Background]
       Groovy's joint compiler will generate a lot of stubs for groovy
source files, many of the stubs are useless and written to disk and clean
later. When project contains a lot of groovy source files, the performance
of compiling is not good.

[Proposal]
      I propose to add an option(e.g. `groovy.generate.stubs.in.memory`) to
generate stubs in in-momery file system[1]. We can get the generated stub
files from the the in-memory file system with `StandardJavaFileManager`[2] .
Here is the `JavaCompiler` usage[3].

[Benefits]
     We can avoid writing lots of stub files to disk and subsequent
cleaning, in addition, reading stub files from memory will be much faster
than reading from disk.


      Any thoughts?

Cheers,
Daniel.Sun
[1] https://github.com/google/jimfs
[2]
https://docs.oracle.com/javase/7/docs/api/javax/tools/StandardJavaFileManager.html
[3]
https://github.com/danielsun1106/SmartASMifier/blob/master/SmartASMifier.groovy#L81-L88




--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance

Daniel.Sun
In reply to this post by Cédric Champeau
Agreed.

I find we can get rid of the 3rd-part lib and just rely on `javax.tools`
APIs, e.g.

```
compilationUnits.add(new MemJavaFileObject(new
URI("string:///labs/Hello.java"), srcStr));
```

I'm doing some experiment.

Cheers,
Daniel.Sun



-----
Apache Groovy committer
Blog: http://blog.sunlan.me 
Twitter: @daniel_sun

--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Apache Groovy committer & PMC member

Blog: http://blog.sunlan.me
Twitter: @daniel_sun
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance

Daniel.Sun
In reply to this post by Daniel Sun
The proposed PR is here: https://issues.apache.org/jira/browse/GROOVY-8962

Merged in 72 hours if no one rejects. Any advice is appreciated!

Cheers,
Daniel.Sun



-----
Apache Groovy committer
Blog: http://blog.sunlan.me 
Twitter: @daniel_sun

--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Apache Groovy committer & PMC member

Blog: http://blog.sunlan.me
Twitter: @daniel_sun
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance

Daniel.Sun
Fix the PR link:  https://github.com/apache/groovy/pull/855



-----
Apache Groovy committer
Blog: http://blog.sunlan.me 
Twitter: @daniel_sun

--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Apache Groovy committer & PMC member

Blog: http://blog.sunlan.me
Twitter: @daniel_sun
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance

Paolo Di Tommaso
Could IDEs like Intellij benefit from this? 


p

On Fri, Jan 18, 2019 at 7:53 PM Daniel.Sun <[hidden email]> wrote:
Fix the PR link:  https://github.com/apache/groovy/pull/855



-----
Apache Groovy committer
Blog: http://blog.sunlan.me
Twitter: @daniel_sun

--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance

Daniel.Sun
If we keep stubs in memory, IDE's incremental compiler can reuse them if no
changes happen.

I think builds of large groovy projects like Grails, Gradle and Groovy
itself, etc will get more benefits from the new feature.


Cheers,
Daniel.Sun




-----
Apache Groovy committer
Blog: http://blog.sunlan.me 
Twitter: @daniel_sun

--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Apache Groovy committer & PMC member

Blog: http://blog.sunlan.me
Twitter: @daniel_sun
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance

Daniel.Sun
In reply to this post by Cédric Champeau
Hi Cédric,

     It seems that gradle has its own java compiler and does not invoke the
following method of groovy, right?

https://github.com/apache/groovy/blob/master/src/main/java/org/codehaus/groovy/tools/javac/JavacJavaCompiler.java#L54

Cheers,
Daniel.Sun



-----
Apache Groovy committer
Blog: http://blog.sunlan.me 
Twitter: @daniel_sun

--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Apache Groovy committer & PMC member

Blog: http://blog.sunlan.me
Twitter: @daniel_sun
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL]Provide an option to generate stubs in in-momery file system for better compiling performance

Jochen Theodorou
In reply to this post by Daniel Sun
The only minus on this is the memory consumption... though I did not
measure it, so maybe it is not that bad.

On 18.01.19 09:44, Daniel Sun wrote:

> Hi all,
>
> [Background]
>         Groovy's joint compiler will generate a lot of stubs for groovy
> source files, many of the stubs are useless and written to disk and clean
> later. When project contains a lot of groovy source files, the performance
> of compiling is not good.
>
> [Proposal]
>        I propose to add an option(e.g. `groovy.generate.stubs.in.memory`) to
> generate stubs in in-momery file system[1]. We can get the generated stub
> files from the the in-memory file system with `StandardJavaFileManager`[2] .
> Here is the `JavaCompiler` usage[3].
>
> [Benefits]
>       We can avoid writing lots of stub files to disk and subsequent
> cleaning, in addition, reading stub files from memory will be much faster
> than reading from disk.
>
>
>        Any thoughts?
>
> Cheers,
> Daniel.Sun
> [1] https://github.com/google/jimfs
> [2]
> https://docs.oracle.com/javase/7/docs/api/javax/tools/StandardJavaFileManager.html
> [3]
> https://github.com/danielsun1106/SmartASMifier/blob/master/SmartASMifier.groovy#L81-L88
>
>
>
>
> --
> Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
>

12