monorepo
Monorepo builds with Gradle and popular CI tools
This is an example of how to manage building projects inside monorepo with Gradle as build tool and one of the following services as CI tool:
Service
Status
GitHub Actions
Motivation
When I push some changes to monorepository I want to
- build only modified projects
- build all other projects depending on modified projects
- build projects in parallel if it is possible
- not build projects when their dependencies are failing
- discover dependencies between projects automatically
How it works
There is only one main job called build started automatically on every push. This job is responsible for triggering another jobs for each affected project in order with respecting project dependencies.
Build job is running until all triggered jobs are finished.
Build job is successful only when there were no failed jobs (even when there were no jobs).
Where are projects defined
There is file tools/ci/projects.txt
which contains lines with glob patterns pointing to root directories of all supported projects.
Where are jobs defined
Jobs are defined in default location depending on which CI tool you are using.
-
CircleCI –
.circleci/config.yml
-
Bitbucket Pipelines –
bitbucket-pipelines.yml
-
Travis CI –
.travis.yml
-
GitHub Actions –
.github/workflows/
How projects are mapped to jobs
Currently there is a convention used for mapping project to CI job. Job name is resolved from project’s directory path as last path component.
e.g. project under directory
apps/server
is built by jobserver
.
How dependencies between projects are resolved
Dependencies are based on Gradle’s composite build feature. To define dependency between projects use includeBuild
function in project build script (usually in settings.gradle
).
If you are not using Gradle or you want to add some additional dependency you can specify it in dependency file. Default location is in each project directory on path .ci/dependencies.txt
. Location can be changed by setting environment variable CI_DEPENDENCIES_FILE
. File should contain lines with paths to other projects (relative paths to monorepo root, e.g. libs/common).
How dependencies affects job triggering
To respect dependencies between projects jobs are triggered in multiple rounds. For each round one or more jobs are triggered and only when all jobs are successfully finished next round is processed. Even if there is only one failed job all next rounds are skipped and whole build is failed.
Special commands
Commit message can contain some special words which if found can modify default building behavior.
Supported commands:
- [rebuild-all] – build all projects instead of only changed
Implementation
Whole logic is implemented as bunch of Bash scripts under directory tools/ci
. Every script can be run directly and it should output some documentation when it requires some parameters.
Implementation is split to core logic and plugins for different CI tools. Core logic can be found under tools/ci/core
and plugins are under tools/ci/plugins
.
Which one is main script
Main script is tools/ci/core/build.sh
and it is only thing started from build job.
Are there any non-standard tools used
There is tool called jq used for JSON parsing.
You need to care only when you want run it locally because jq is usually part of default images used in CI tools.
How to run locally
It is possible to run tools/ci/core/build.sh
locally but there is need to provide few environment variables depending on CI tool used.
CircleCI
CI_TOOL=circleci
CIRCLE_API_USER_TOKEN=XXX
CIRCLE_PROJECT_USERNAME=zladovan
CIRCLE_PROJECT_REPONAME=monorepo
CIRCLE_BRANCH=master
CIRCLE_SHA1=$(git rev-parse HEAD)
tools/ci/core/build.sh
Where:
- CIRCLE_API_USER_TOKEN is your private token
- CIRCLE_SHA1 should be set to current commit hash, but it can be any commit hash
Note that this command could trigger some jobs in circleci
Bitbucket pipelines
CI_TOOL=bitbucket
BITBUCKET_USER=zladovan
BITBUCKET_PASSWORD=xxx
BITBUCKET_REPO_FULL_NAME=zladovan/monorepo
BITBUCKET_BRANCH=master
BITBUCKET_COMMIT=$(git rev-parse HEAD)
tools/ci/core/build.sh
Where:
- BITBUCKET_COMMIT should be set to current commit hash, but it can be any commit hash
Note that this command could trigger some jobs in bitbucket
Travis CI
CI_TOOL=travis
TRAVIS_TOKEN=xxx
TRAVIS_REPO_SLUG=zladovan/monorepo
TRAVIS_BRANCH=master
TRAVIS_COMMIT=$(git rev-parse HEAD)
tools/ci/core/build.sh
Where:
- TRAVIS_COMMIT should be set to current commit hash, but it can be any commit hash
Note that this command could trigger some jobs in travis
GitHub Actions
CI_TOOL=github
GITHUB_REPOSITORY=zladovan/monorepo
GITHUB_TOKEN=xxx
GITHUB_REF=refs/head/master
GITHUB_SHA=$(git rev-parse HEAD)
tools/ci/core/build.sh
Where:
- GITHUB_TOKEN should be your personal access token with repo rights
- GITHUB_SHA should be set to current commit hash, but it can be any commit hash
Note that this command could trigger some jobs in github
Folder structure
Folder structure used in this repository is only an example of how it can look like. It is possible to use any structure, there is only need to use different patterns in tools/ci/projects.txt
apps/
└── stand-alone runnable and deployable applications
libs/
└── reusable libraries (used in apps dependencies)
tools/gradle-plugins/
└── reusable gradle logic (used in apps and libs builds)
tools/ci
└── ci scripts
Known issues
- jobs are not triggered in parallel for all cases due to using tsort for processing dependencies which produce only sequential order
-
not tested on Mac OS and probably there will be issue with
realpath
used - Travis CI has some limitations around how many builds can be started via API
Todo
- write how to setup for different ci tools
- improve parallel executions support
- add special command to trigger build for specific project
- create Gradle plugin with same logic as in bash scripts (as a separate project)
- add support for other popular CI tools (e.g. Jenkins, …)
- create downloadable package which could be installed to repo during build time
- create Circleci orb
- create GitHub Action
Credits
Thanks to Tufin for inspiration in Tufin/circleci-monorepo.