[jira] [Comment Edited] (GROOVY-8097) Add an argument to set the resolution cache path in @Grab

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[jira] [Comment Edited] (GROOVY-8097) Add an argument to set the resolution cache path in @Grab

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/GROOVY-8097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15919911#comment-15919911 ]

Ion Alberdi edited comment on GROOVY-8097 at 3/14/17 8:34 AM:
--------------------------------------------------------------

[~jexler]
"Assuming that using different GrapeIvy instances in parallel would be thread-safe (which I do not know if it is the case)"
The script [grab.groovy |https://github.com/yetanotherion/code_snippets/blob/master/groovy/groovy_8097/grab.groovy] uses 200 different grapeIvy instances in parallel successfully.

"dir-with-random-name"
- Roughly speaking, we have at least 200 jobs running every 5 minutes, each making at least two calls to Grab in the jenkins master. If each grab uses its own grapeIvy instance creating a random resolution dir, we'll end up with 19200 directories in 8 hours in the jenkins master.
- Moreover, as said above, the dir-with-random-name solution is still (even if very unlikely) prone to collisions, when the static solution solves it (providing the developer does not make any mistake).

I'm afraid that this strategy is not adapted to our use case. I agree that the per-concurrent job strategy asks much more to the developer, but it permits to handle such a load (as the number of concurrent jobs does not increase linearly overtime).

That does not prevent adding another strategy for different use cases though.





was (Author: yetanotherion):
[~jexler]
"Assuming that using different GrapeIvy instances in parallel would be thread-safe (which I do not know if it is the case)"
The script [grab.groovy |https://github.com/yetanotherion/code_snippets/blob/master/groovy/groovy_8097/grab.groovy] above uses 200 different grapeIvy instances in parallel successfully.

"dir-with-random-name"
- Roughly speaking, we have at least 200 jobs running every 5 minutes, each making at least two calls to Grab in the jenkins master. If each grab uses its own grapeIvy instance creating a random resolution dir, we'll end up with 19200 directories in 8 hours in the jenkins master.
- Moreover, as said above, the dir-with-random-name solution is still (even if very unlikely) prone to collisions, when the static solution solves it (providing the developer does not make any mistake).

I'm afraid that this strategy is not adapted to our use case. I agree that the per-concurrent job strategy asks much more to the developer, but it permits to handle such a load (as the number of concurrent jobs does not increase linearly overtime).

That does not prevent adding another strategy for different use cases though.




> Add an argument to set the resolution cache path in @Grab
> ---------------------------------------------------------
>
>                 Key: GROOVY-8097
>                 URL: https://issues.apache.org/jira/browse/GROOVY-8097
>             Project: Groovy
>          Issue Type: New Feature
>          Components: Grape
>    Affects Versions: 2.4.8
>            Reporter: Ion Alberdi
>            Priority: Minor
>
> Ivy does not support concurrent access to its resolution cache
> https://issues.apache.org/jira/browse/IVY-654
> Grape relies on Ivy. For this reason, Grape cannot support concurrent access to its resolution cache neither.
> When using the @Grab annotation in jenkins groovyCommand or systemGroovyCommand, the related code is vulnerable to race conditions. When the race condition appears in a systemGroovyCommand, we have no choice but to reboot jenkins as all consecutive calls to @Grab fail.
> Among the two solutions we tried:
> - Protect the calls to grab with a lock similar to ivy's "artifact-lock-nio" strategy. Works but slow.
> - Set Ivy's lock on the repository cache and setup Grab to use a different cache resolution cache for each concurrent jobs. The following code permits to fix a test we did to reproduce the race condition.
> {code}
>     static IvySettings createIvySettings(String resolutionPath, boolean dumpSettings) {
>         // Copy/Paste/Purged from GrapeIvy.groovy
>         IvySettings settings = new IvySettings()
>         settings.load(new File(GROOVY_HOME, "grapeConfig.xml"))
>         // set up the cache dirs
>         settings.defaultCache = new File(GRAPES_HOME)
>         settings.setVariable("ivy.default.configuration.m2compatible", "true")
>         settings.setDefaultResolutionCacheBasedir(resolutionPath)
>         return settings
>     }
>     static GrapeIvy ivyWithCustomResolutionPath(String resolutionPath) {
>         Class<?> grapeIvyClass = Class.forName("groovy.grape.GrapeIvy");
>         Object instance = grapeIvyClass.newInstance()
>         Field field = grapeIvyClass.getDeclaredField("ivyInstance");
>         field.setAccessible(true);
>         field.set(instance, Ivy.newInstance(createIvySettings(resolutionPath)));
>         return ((GrapeIvy)instance)
>     }
> {code}
> We'd like to propose to add an additional argument to Grab to setup Ivy's resolution cache directory.
> Note that this solution seems to have been adopted by these users too
> https://rbcommons.com/s/twitter/r/3436/
> Would you agree on such a feature ? We'd be glad to propose a PR.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Loading...