Groovy and code coverage

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Groovy and code coverage

Andres Almiray
Hello everyone,

As you may be aware there are only a few choices in the Java pace that can be used for code coverage, JaCoCo being the one that gives the most accurate results. While it's easy to setup JaCoco for Groovy sources there are times where the tool reports no coverage for some lines of code that should not have been counted for coverage in the first case, these are some of the methods coming from GroovyObject and AST transformations, specifically `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod`.

I had a chat with Marc Hoffmann yesterday about this situation, which is a topic we've discussed a few times in the past. I've got some good news, Groovy is not alone in this problem, the Lombok project has suffered the same fate which has prompted them to seek a solution. The alternative they came up with is to have Lombok identified generated code that should not be covered with a special annotation (@lombok.Generated), JaCoCo has a filtering feature (not yet public) that can identify elements annotated with said annotation and skip them from coverage reports. See

https://projectlombok.org/api/index.html?lombok/Generated.html
https://github.com/jacoco/jacoco/pull/513
https://github.com/jacoco/jacoco/blob/master/org.jacoco.core/src/org/jacoco/core/internal/analysis/filter/LombokGeneratedFilter.java

This feature will be available for consumption in the next releases of Lombok (1.16.8+) and JaCoCo (0.7.9+).

Given this, Marc suggested that Groovy could follow the same idea and provide an annotation that JaCoCo can identify, say for example @groovy.transform.Generated.

If such annotation were to be added to Groovy we would need (at least) the following steps too:
 - update compiler to annotate `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod` when the class does not explicitly defines any of those methods.
 - review and update all core AST xforms that generate methods that should be skipped from coverage.
 - document the usage of this annotation and encourage AST xform developers to use it once they upgrade.

Would be great to have this feature in the 2.6.0 release if possible.

Thoughts?

-------------------------------------------
Java Champion; Groovy Enthusiast
http://andresalmiray.com
http://www.linkedin.com/in/aalmiray
--
What goes up, must come down. Ask any system administrator.
There are 10 types of people in the world: Those who understand binary, and those who don't.
To understand recursion, we must first understand recursion.
Reply | Threaded
Open this post in threaded view
|

Re: Groovy and code coverage

Cédric Champeau
I'm not super keen on adding a "feature" like that. First, we already mark generated methods, thanks to the "synthetic" modifier that fields, methods and classes can have. Second, either we would have to change the code of all AST transformations to add this at the relevant places, or automatically do it. Unfortunately, automatic is not possible, because it is very well possible for an AST transformation to generate code which is _actually found in the sources_ (duplicate AST, rename a method, ...).

Last, strictly speaking I don't see why we wouldn't count invokeMethod and friends into coverage. They _effectively_ participate in coverage.

2017-09-08 11:41 GMT+02:00 Andres Almiray <[hidden email]>:
Hello everyone,

As you may be aware there are only a few choices in the Java pace that can be used for code coverage, JaCoCo being the one that gives the most accurate results. While it's easy to setup JaCoco for Groovy sources there are times where the tool reports no coverage for some lines of code that should not have been counted for coverage in the first case, these are some of the methods coming from GroovyObject and AST transformations, specifically `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod`.

I had a chat with Marc Hoffmann yesterday about this situation, which is a topic we've discussed a few times in the past. I've got some good news, Groovy is not alone in this problem, the Lombok project has suffered the same fate which has prompted them to seek a solution. The alternative they came up with is to have Lombok identified generated code that should not be covered with a special annotation (@lombok.Generated), JaCoCo has a filtering feature (not yet public) that can identify elements annotated with said annotation and skip them from coverage reports. See

https://projectlombok.org/api/index.html?lombok/Generated.html
https://github.com/jacoco/jacoco/pull/513
https://github.com/jacoco/jacoco/blob/master/org.jacoco.core/src/org/jacoco/core/internal/analysis/filter/LombokGeneratedFilter.java

This feature will be available for consumption in the next releases of Lombok (1.16.8+) and JaCoCo (0.7.9+).

Given this, Marc suggested that Groovy could follow the same idea and provide an annotation that JaCoCo can identify, say for example @groovy.transform.Generated.

If such annotation were to be added to Groovy we would need (at least) the following steps too:
 - update compiler to annotate `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod` when the class does not explicitly defines any of those methods.
 - review and update all core AST xforms that generate methods that should be skipped from coverage.
 - document the usage of this annotation and encourage AST xform developers to use it once they upgrade.

Would be great to have this feature in the 2.6.0 release if possible.

Thoughts?

-------------------------------------------
Java Champion; Groovy Enthusiast
http://andresalmiray.com
http://www.linkedin.com/in/aalmiray
--
What goes up, must come down. Ask any system administrator.
There are 10 types of people in the world: Those who understand binary, and those who don't.
To understand recursion, we must first understand recursion.

Reply | Threaded
Open this post in threaded view
|

Re: Groovy and code coverage

Remi Forax
In reply to this post by Andres Almiray
A standard annotation for generated code was added in Java 9:
http://download.java.net/java/jdk9/docs/api/javax/annotation/processing/Generated.html

Rémi


De: "Andres Almiray" <[hidden email]>
À: [hidden email]
Envoyé: Vendredi 8 Septembre 2017 11:41:18
Objet: Groovy and code coverage
Hello everyone,

As you may be aware there are only a few choices in the Java pace that can be used for code coverage, JaCoCo being the one that gives the most accurate results. While it's easy to setup JaCoco for Groovy sources there are times where the tool reports no coverage for some lines of code that should not have been counted for coverage in the first case, these are some of the methods coming from GroovyObject and AST transformations, specifically `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod`.

I had a chat with Marc Hoffmann yesterday about this situation, which is a topic we've discussed a few times in the past. I've got some good news, Groovy is not alone in this problem, the Lombok project has suffered the same fate which has prompted them to seek a solution. The alternative they came up with is to have Lombok identified generated code that should not be covered with a special annotation (@lombok.Generated), JaCoCo has a filtering feature (not yet public) that can identify elements annotated with said annotation and skip them from coverage reports. See

https://projectlombok.org/api/index.html?lombok/Generated.html
https://github.com/jacoco/jacoco/pull/513
https://github.com/jacoco/jacoco/blob/master/org.jacoco.core/src/org/jacoco/core/internal/analysis/filter/LombokGeneratedFilter.java

This feature will be available for consumption in the next releases of Lombok (1.16.8+) and JaCoCo (0.7.9+).

Given this, Marc suggested that Groovy could follow the same idea and provide an annotation that JaCoCo can identify, say for example @groovy.transform.Generated.

If such annotation were to be added to Groovy we would need (at least) the following steps too:
 - update compiler to annotate `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod` when the class does not explicitly defines any of those methods.
 - review and update all core AST xforms that generate methods that should be skipped from coverage.
 - document the usage of this annotation and encourage AST xform developers to use it once they upgrade.

Would be great to have this feature in the 2.6.0 release if possible.

Thoughts?

-------------------------------------------
Java Champion; Groovy Enthusiast
http://andresalmiray.com
http://www.linkedin.com/in/aalmiray
--
What goes up, must come down. Ask any system administrator.
There are 10 types of people in the world: Those who understand binary, and those who don't.
To understand recursion, we must first understand recursion.

Reply | Threaded
Open this post in threaded view
|

Re: Groovy and code coverage

Cédric Champeau
Damn, who thought it was a good idea to put TIMESTAMPS in generated sources?

2017-09-08 15:00 GMT+02:00 Remi Forax <[hidden email]>:
A standard annotation for generated code was added in Java 9:

Rémi


De: "Andres Almiray" <[hidden email]>
À: [hidden email]
Envoyé: Vendredi 8 Septembre 2017 11:41:18
Objet: Groovy and code coverage
Hello everyone,

As you may be aware there are only a few choices in the Java pace that can be used for code coverage, JaCoCo being the one that gives the most accurate results. While it's easy to setup JaCoco for Groovy sources there are times where the tool reports no coverage for some lines of code that should not have been counted for coverage in the first case, these are some of the methods coming from GroovyObject and AST transformations, specifically `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod`.

I had a chat with Marc Hoffmann yesterday about this situation, which is a topic we've discussed a few times in the past. I've got some good news, Groovy is not alone in this problem, the Lombok project has suffered the same fate which has prompted them to seek a solution. The alternative they came up with is to have Lombok identified generated code that should not be covered with a special annotation (@lombok.Generated), JaCoCo has a filtering feature (not yet public) that can identify elements annotated with said annotation and skip them from coverage reports. See

https://projectlombok.org/api/index.html?lombok/Generated.html
https://github.com/jacoco/jacoco/pull/513
https://github.com/jacoco/jacoco/blob/master/org.jacoco.core/src/org/jacoco/core/internal/analysis/filter/LombokGeneratedFilter.java

This feature will be available for consumption in the next releases of Lombok (1.16.8+) and JaCoCo (0.7.9+).

Given this, Marc suggested that Groovy could follow the same idea and provide an annotation that JaCoCo can identify, say for example @groovy.transform.Generated.

If such annotation were to be added to Groovy we would need (at least) the following steps too:
 - update compiler to annotate `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod` when the class does not explicitly defines any of those methods.
 - review and update all core AST xforms that generate methods that should be skipped from coverage.
 - document the usage of this annotation and encourage AST xform developers to use it once they upgrade.

Would be great to have this feature in the 2.6.0 release if possible.

Thoughts?

-------------------------------------------
Java Champion; Groovy Enthusiast
http://andresalmiray.com
http://www.linkedin.com/in/aalmiray
--
What goes up, must come down. Ask any system administrator.
There are 10 types of people in the world: Those who understand binary, and those who don't.
To understand recursion, we must first understand recursion.


Reply | Threaded
Open this post in threaded view
|

Re: Groovy and code coverage

Andres Almiray
In reply to this post by Remi Forax
Unfortunately that annotation is coming in JDK9. We'll need something that can be used with JDK7
 an upwards.

-------------------------------------------
Java Champion; Groovy Enthusiast
http://andresalmiray.com
http://www.linkedin.com/in/aalmiray
--
What goes up, must come down. Ask any system administrator.
There are 10 types of people in the world: Those who understand binary, and those who don't.
To understand recursion, we must first understand recursion.

On Fri, Sep 8, 2017 at 3:00 PM, Remi Forax <[hidden email]> wrote:
A standard annotation for generated code was added in Java 9:

Rémi


De: "Andres Almiray" <[hidden email]>
À: [hidden email]
Envoyé: Vendredi 8 Septembre 2017 11:41:18
Objet: Groovy and code coverage
Hello everyone,

As you may be aware there are only a few choices in the Java pace that can be used for code coverage, JaCoCo being the one that gives the most accurate results. While it's easy to setup JaCoco for Groovy sources there are times where the tool reports no coverage for some lines of code that should not have been counted for coverage in the first case, these are some of the methods coming from GroovyObject and AST transformations, specifically `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod`.

I had a chat with Marc Hoffmann yesterday about this situation, which is a topic we've discussed a few times in the past. I've got some good news, Groovy is not alone in this problem, the Lombok project has suffered the same fate which has prompted them to seek a solution. The alternative they came up with is to have Lombok identified generated code that should not be covered with a special annotation (@lombok.Generated), JaCoCo has a filtering feature (not yet public) that can identify elements annotated with said annotation and skip them from coverage reports. See

https://projectlombok.org/api/index.html?lombok/Generated.html
https://github.com/jacoco/jacoco/pull/513
https://github.com/jacoco/jacoco/blob/master/org.jacoco.core/src/org/jacoco/core/internal/analysis/filter/LombokGeneratedFilter.java

This feature will be available for consumption in the next releases of Lombok (1.16.8+) and JaCoCo (0.7.9+).

Given this, Marc suggested that Groovy could follow the same idea and provide an annotation that JaCoCo can identify, say for example @groovy.transform.Generated.

If such annotation were to be added to Groovy we would need (at least) the following steps too:
 - update compiler to annotate `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod` when the class does not explicitly defines any of those methods.
 - review and update all core AST xforms that generate methods that should be skipped from coverage.
 - document the usage of this annotation and encourage AST xform developers to use it once they upgrade.

Would be great to have this feature in the 2.6.0 release if possible.

Thoughts?

-------------------------------------------
Java Champion; Groovy Enthusiast
http://andresalmiray.com
http://www.linkedin.com/in/aalmiray
--
What goes up, must come down. Ask any system administrator.
There are 10 types of people in the world: Those who understand binary, and those who don't.
To understand recursion, we must first understand recursion.


Reply | Threaded
Open this post in threaded view
|

Re: Groovy and code coverage

Remi Forax
You are generating bytecode ?

Rémi


De: "Andres Almiray" <[hidden email]>
À: "dev" <[hidden email]>
Envoyé: Vendredi 8 Septembre 2017 15:04:57
Objet: Re: Groovy and code coverage
Unfortunately that annotation is coming in JDK9. We'll need something that can be used with JDK7
 an upwards.

-------------------------------------------
Java Champion; Groovy Enthusiast
http://andresalmiray.com
http://www.linkedin.com/in/aalmiray
--
What goes up, must come down. Ask any system administrator.
There are 10 types of people in the world: Those who understand binary, and those who don't.
To understand recursion, we must first understand recursion.

On Fri, Sep 8, 2017 at 3:00 PM, Remi Forax <[hidden email]> wrote:
A standard annotation for generated code was added in Java 9:

Rémi


De: "Andres Almiray" <[hidden email]>
À: [hidden email]
Envoyé: Vendredi 8 Septembre 2017 11:41:18
Objet: Groovy and code coverage
Hello everyone,

As you may be aware there are only a few choices in the Java pace that can be used for code coverage, JaCoCo being the one that gives the most accurate results. While it's easy to setup JaCoco for Groovy sources there are times where the tool reports no coverage for some lines of code that should not have been counted for coverage in the first case, these are some of the methods coming from GroovyObject and AST transformations, specifically `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod`.

I had a chat with Marc Hoffmann yesterday about this situation, which is a topic we've discussed a few times in the past. I've got some good news, Groovy is not alone in this problem, the Lombok project has suffered the same fate which has prompted them to seek a solution. The alternative they came up with is to have Lombok identified generated code that should not be covered with a special annotation (@lombok.Generated), JaCoCo has a filtering feature (not yet public) that can identify elements annotated with said annotation and skip them from coverage reports. See

https://projectlombok.org/api/index.html?lombok/Generated.html
https://github.com/jacoco/jacoco/pull/513
https://github.com/jacoco/jacoco/blob/master/org.jacoco.core/src/org/jacoco/core/internal/analysis/filter/LombokGeneratedFilter.java

This feature will be available for consumption in the next releases of Lombok (1.16.8+) and JaCoCo (0.7.9+).

Given this, Marc suggested that Groovy could follow the same idea and provide an annotation that JaCoCo can identify, say for example @groovy.transform.Generated.

If such annotation were to be added to Groovy we would need (at least) the following steps too:
 - update compiler to annotate `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod` when the class does not explicitly defines any of those methods.
 - review and update all core AST xforms that generate methods that should be skipped from coverage.
 - document the usage of this annotation and encourage AST xform developers to use it once they upgrade.

Would be great to have this feature in the 2.6.0 release if possible.

Thoughts?

-------------------------------------------
Java Champion; Groovy Enthusiast
http://andresalmiray.com
http://www.linkedin.com/in/aalmiray
--
What goes up, must come down. Ask any system administrator.
There are 10 types of people in the world: Those who understand binary, and those who don't.
To understand recursion, we must first understand recursion.


Reply | Threaded
Open this post in threaded view
|

Re: Groovy and code coverage

Remi Forax
In reply to this post by Cédric Champeau



De: "Cédric Champeau" <[hidden email]>
À: "dev" <[hidden email]>
Envoyé: Vendredi 8 Septembre 2017 15:03:05
Objet: Re: Groovy and code coverage
Damn, who thought it was a good idea to put TIMESTAMPS in generated sources?

It's a drop in replacement for javax.annotation.Generated [1] which is in a module what will be removed [2] from the modules of Java SE,
so a long time ago :)

Rémi

[1] https://docs.oracle.com/javase/8/docs/api/javax/annotation/Generated.html
[2] https://bugs.openjdk.java.net/browse/JDK-8152842


2017-09-08 15:00 GMT+02:00 Remi Forax <[hidden email]>:
A standard annotation for generated code was added in Java 9:

Rémi


De: "Andres Almiray" <[hidden email]>
À: [hidden email]
Envoyé: Vendredi 8 Septembre 2017 11:41:18
Objet: Groovy and code coverage
Hello everyone,

As you may be aware there are only a few choices in the Java pace that can be used for code coverage, JaCoCo being the one that gives the most accurate results. While it's easy to setup JaCoco for Groovy sources there are times where the tool reports no coverage for some lines of code that should not have been counted for coverage in the first case, these are some of the methods coming from GroovyObject and AST transformations, specifically `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod`.

I had a chat with Marc Hoffmann yesterday about this situation, which is a topic we've discussed a few times in the past. I've got some good news, Groovy is not alone in this problem, the Lombok project has suffered the same fate which has prompted them to seek a solution. The alternative they came up with is to have Lombok identified generated code that should not be covered with a special annotation (@lombok.Generated), JaCoCo has a filtering feature (not yet public) that can identify elements annotated with said annotation and skip them from coverage reports. See

https://projectlombok.org/api/index.html?lombok/Generated.html
https://github.com/jacoco/jacoco/pull/513
https://github.com/jacoco/jacoco/blob/master/org.jacoco.core/src/org/jacoco/core/internal/analysis/filter/LombokGeneratedFilter.java

This feature will be available for consumption in the next releases of Lombok (1.16.8+) and JaCoCo (0.7.9+).

Given this, Marc suggested that Groovy could follow the same idea and provide an annotation that JaCoCo can identify, say for example @groovy.transform.Generated.

If such annotation were to be added to Groovy we would need (at least) the following steps too:
 - update compiler to annotate `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod` when the class does not explicitly defines any of those methods.
 - review and update all core AST xforms that generate methods that should be skipped from coverage.
 - document the usage of this annotation and encourage AST xform developers to use it once they upgrade.

Would be great to have this feature in the 2.6.0 release if possible.

Thoughts?

-------------------------------------------
Java Champion; Groovy Enthusiast
http://andresalmiray.com
http://www.linkedin.com/in/aalmiray
--
What goes up, must come down. Ask any system administrator.
There are 10 types of people in the world: Those who understand binary, and those who don't.
To understand recursion, we must first understand recursion.


Reply | Threaded
Open this post in threaded view
|

Re: Groovy and code coverage

Andres Almiray
I see. Also, the retention policy JaCoCo expects is CLASS, not SOURCE.

-------------------------------------------
Java Champion; Groovy Enthusiast
http://andresalmiray.com
http://www.linkedin.com/in/aalmiray
--
What goes up, must come down. Ask any system administrator.
There are 10 types of people in the world: Those who understand binary, and those who don't.
To understand recursion, we must first understand recursion.

On Fri, Sep 8, 2017 at 3:11 PM, Remi Forax <[hidden email]> wrote:



De: "Cédric Champeau" <[hidden email]>
À: "dev" <[hidden email]>
Envoyé: Vendredi 8 Septembre 2017 15:03:05
Objet: Re: Groovy and code coverage
Damn, who thought it was a good idea to put TIMESTAMPS in generated sources?

It's a drop in replacement for javax.annotation.Generated [1] which is in a module what will be removed [2] from the modules of Java SE,
so a long time ago :)

Rémi



2017-09-08 15:00 GMT+02:00 Remi Forax <[hidden email]>:
A standard annotation for generated code was added in Java 9:

Rémi


De: "Andres Almiray" <[hidden email]>
À: [hidden email]
Envoyé: Vendredi 8 Septembre 2017 11:41:18
Objet: Groovy and code coverage
Hello everyone,

As you may be aware there are only a few choices in the Java pace that can be used for code coverage, JaCoCo being the one that gives the most accurate results. While it's easy to setup JaCoco for Groovy sources there are times where the tool reports no coverage for some lines of code that should not have been counted for coverage in the first case, these are some of the methods coming from GroovyObject and AST transformations, specifically `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod`.

I had a chat with Marc Hoffmann yesterday about this situation, which is a topic we've discussed a few times in the past. I've got some good news, Groovy is not alone in this problem, the Lombok project has suffered the same fate which has prompted them to seek a solution. The alternative they came up with is to have Lombok identified generated code that should not be covered with a special annotation (@lombok.Generated), JaCoCo has a filtering feature (not yet public) that can identify elements annotated with said annotation and skip them from coverage reports. See

https://projectlombok.org/api/index.html?lombok/Generated.html
https://github.com/jacoco/jacoco/pull/513
https://github.com/jacoco/jacoco/blob/master/org.jacoco.core/src/org/jacoco/core/internal/analysis/filter/LombokGeneratedFilter.java

This feature will be available for consumption in the next releases of Lombok (1.16.8+) and JaCoCo (0.7.9+).

Given this, Marc suggested that Groovy could follow the same idea and provide an annotation that JaCoCo can identify, say for example @groovy.transform.Generated.

If such annotation were to be added to Groovy we would need (at least) the following steps too:
 - update compiler to annotate `getProperty`, `setProperty`, `getMetaClass`, and `invokeMethod` when the class does not explicitly defines any of those methods.
 - review and update all core AST xforms that generate methods that should be skipped from coverage.
 - document the usage of this annotation and encourage AST xform developers to use it once they upgrade.

Would be great to have this feature in the 2.6.0 release if possible.

Thoughts?

-------------------------------------------
Java Champion; Groovy Enthusiast
http://andresalmiray.com
http://www.linkedin.com/in/aalmiray
--
What goes up, must come down. Ask any system administrator.
There are 10 types of people in the world: Those who understand binary, and those who don't.
To understand recursion, we must first understand recursion.



Reply | Threaded
Open this post in threaded view
|

Re: Groovy and code coverage

Jochen Theodorou
In reply to this post by Cédric Champeau
On 08.09.2017 11:55, Cédric Champeau wrote:
> I'm not super keen on adding a "feature" like that. First, we already
> mark generated methods, thanks to the "synthetic" modifier that fields,
> methods and classes can have.

synthetic means for an IDE to not to show that method. That may not be
right for things like getProperty, setProperty and getMetaClass. So for
me it makes sense to have non-synthetic code marked like that.

bye Jochen

Reply | Threaded
Open this post in threaded view
|

Re: Groovy and code coverage

Jochen Theodorou
In reply to this post by Andres Almiray
On 08.09.2017 15:04, Andres Almiray wrote:
> Unfortunately that annotation is coming in JDK9. We'll need something
> that can be used with JDK7

actually that would not be a problem. If the annotation is not found it
is ignored by the java compiler. The source retention policy though is a
very good reason not to use it. So we have to roll our own.

I am +1 on the idea in general.... doesn´t mean all transforms have to
do it right away.

bye Jochen


12