Long lived caches of several of my projects grow with time, reaching unacceptable levels, such as 40GB. Cronjobs prevent caches from autocleaning (and cronjobs must remain there).
I’d like a way to automatically clean a cache over a certain size. Say, for instance, that I know that if it gets past 10GB, then it accumulated so many old (and unused) packages that I’d like to clean it.
Is there a way already?
I believe a max_size key under cache would be great.
Nothing prevents you from doing this in the build in the before_cache: phase. This phase is specifically intended to clean up the assets to be cached.
This suggests to me that there is something bad about what you are caching. Some utilities may be caching more things aggressively than they should (or we are caching too much). If you can identify what is causing this unnatural growth, I think we can all be happier.
Sure, but a build in check would be much better than parsing the output of du. Or is there a better way?
Also, I’m not sure how portable it is, and I’m on a multi-OS build.
I think I clean it quite neatly via Gravis CI. The thing is that the project’s got many libraries, and they are update frequently, so the cache expands a little bit on every update.
plus, it’s a multi-OS, multi-JDK build with 4-19 jobs, so if every job has a 2GB cache it’s easy to get to 40GB overall for a branch.
The output of du is specifically designed to be machine-readable. Each line starts with a number (assuming -m), then a tab.
^(\d+)\t regex (or equivalent).-c
du -smxc * | tail -n 1 | cut -f 1 will give you the raw number in MiBdu -smxc is available in all 3 OSes. -0 is not available in OSX, but I guess you probably don’t have files with newlines in names (to get a du with one, install the coreutils Homebrew package. It will be available as gdu, or as du (overriding the stock one) if you also add to PATH according to Homebrew’s instructions).
If this is the case, you don’t actually “clean it quite neatly”. You want something akin to brew cleanup which checks actual metadata about what’s in use (or make some other tool used in the build produce some kind of trace info about what’s in use, then use the resulting list).
I disagree. There’s no way in Gradle to do what you suggest. It caches all the libraries in the user home, and you got no clue which are used in which phase.
Or actually, you could ask the tool, and then write a software that does the inspection for you, and that would be way more expensive than just let the cache expand until the build fails and then clean it up manually for a hundred years.
Anyway, if you got another suggestion similar to the majestic one on the use of du that also cleans the gradle cache selectively, I’d be glad to learn.
Cheers
Googled on “clean up gradle cache of unused packages”:
Thank you for the syntactic match on a search engine.
The first one is interesting, but if it works automatically, then my caches should be already cleaned up by the daemon (in addition to the cleaning I perform to prevent the cache from being repackaged at every run). I may force a --stop in before_cache and see if it improves - maybe the deamon doesn’t get shut down correctly automatically.
Unfortunately, the latter two links you posted are on entirely another topic. Pruning unused dependencies in a single project does not mean cleaning up the dependency cache from the home folder of a system. It is entirely different (and dependencies are regularly maintained).
Let me explain: every Gradle run caches the downloaded jars (and other stuff that I delete already) in the home folder.
If I use dependency X 1.0, it gets cached.
If I upgrade to X 2.0, it get cached as well, but X 1.0 does not get deleted as Gradle has no mean to know it is not used by another project, or still by the same project in a different lifecycle phase – but to wait and see if it has never got accessed for a long time (which looks like being the strategy of the first link).
A proactive search of unused cached libraries (not dependencies to prune) would require a new tool, AFAIK.