Motivation

Gradle projects usually have a settings.gradle file, which defines the Gradle modules to be included in the build and eventually other stuff that must happen before the project’s configuration phase, like plugins management or Composite builds. Can we prevent this file from becoming a mess in a large project hosted in a monorepo?

By large monorepo here, let’s consider a hypothetical example of a project with circa 750 000 lines of code (production and tests) distributed across 150 Gradle modules. The project structure is more or less well organised (with room for improvements as always) and it promotes a good separation between features code and infrastructure-for-features code at the project tree-level:

  .
  ├── app
  ├── buildSrc
  ├── core
  │   ├── navigation
  │   ├── notifications
  │   ├── sms
  │   ├── ...
  │   └── utils
  ├── ...
  ├── ...
  ├── features
  │   ├── login
  │   ├── home
  │   ├── ...
  │   └── checkout
  │
  └── settings.gradle

For sure there are Gradle projects in monorepos that are BIG and don’t meet these assumptions, but this is not the point here; the real thing is that we can spot a few common problems that arise from such a type of project.

A big settings.gradle file

It is easy to realise that, with 150 Gradle modules, settings.gradle or settings.gradle.kts file will contains at least 150 lines like

Also, it is possible that more pre-configuration logic needs to live in settings.gradle as well. We can take as an example the settings file from Gradle itself, which has a block for plugins management and others.

Pain with module’s names management

Let’s say that we identify some good library candidate inside one of the product feature Gradle submodules. We want to move this library upwards in the project tree. This means that we need to refactor both the include statement in settings.gradle and also the dependencies configuration in the consumer modules

One situation that brings pain with name management for such a project happens when moving Gradle modules. Unfortunately, moving Gradle modules inside a big project structure like the one aforementioned is usually a pain and unfortunately IDE refactor does not help that much here (which is sad).

One common strategy to smooth this problem a bit is statically mapping the Gradle coordinates for module names as Strings living in buildSrc

Since this object will become project-wide available, at least when refactoring we just need to change one (1) string reference and re-sync the project. The issue here is : taking in consideration our hypothetical example, we now would have 150 lines of includes in settings and another 150 lines of mapping in GradleModules.kt, so for sure there is a trade-off here since ideally these two files must live in sync somehow.

Magic Modules

Usually Gradle docs and examples for plugin authors focus on plugins driven by the Project Gradle API :

However, we can also have plugins that hook custom build logic at the Settings instance.

The Settings Gradle API allows us to do several things and for us, the eye-catching functionally here is the possibility to add Gradle modules dynamically.

Therefore, what if we write a Gradle plugin for Settings that

  1. Walks the project tree and find all applicable build.gradle and build.gradle.kts files
  2. Parses such files in order to map if they match (JVM|Android) x (library|application) module
  3. Uses the mapped information to add all needed Gradle modules at plugin execution time and also write a file like ModuleNames.kt under buildSrc

Well, this is exactly what I tried with Magic Modules, an experimental Gradle plugin that tackles the problem described in this article. The plugin code is quite straightforward to follow, despite some logging and Plugin Extensions that I added later

You can learn on Github instructions about how to try this plugin. Some implementation details that are interesting to highlight here, though :

  • Project walking is handled by a recursive deep-first search, which is OK for parsing hundreds of eventually a few thousand of nodes in the project tree
  • The proof of concept is well tested with integration tests that exercise several corner case scenarios like empty Gradle projects and projects without buildSrc. Such tests were written with Gradle TestKit
  • KotlinPoet was used to write constants on Libraries.kt and Applications.kt files

I decided to split the generation of constants under buildSrc into two files, because I’ve seen that multi-module monorepos often rely on launcher apps for each particular feature in order to improve the build times and developer experience when working with code at features level. This is what happens at N26, for instance. The plugin figures out the right name of the constant based on the mapped Gradle coordinates, and generates a nice file like this one :

When working with the launcher apps project pattern usually each developer works only at one feature at time, which means that makes no-sense to have all launcher apps plus the monolithic one (that will be AppBundled to Google Play) included in the build all the time, either locally and eventually at CI as well. MagicModules proposes an entry point to tackle that with a particular plugin extension as well :

Conclusions

Large projects in monorepos are full of Engineering challenges, especially when working with advanced and complex build systems like Gradle. Eventually the Gradle’s project structure is tailored to mirror somehow the organization of product teams, and since organizations change all time most likely Gradle’s project structure continues to change and evolve as well, something that might bring pain even in the little details like the project settings.

MagicModules is an attempt to solve one of those little points of pain. It is fully experimental and it has some limitations - for instance, the code generated under buildSrc as part of the plugin is not ktlinted at all, and I’m not sure how the Gradle IDE model is affected by it - and I’m sharing it just as a proof of concept. I would love to learn your thoughts about.