Introducing Zim: A caching build system for teams using monorepos

Zim is a caching build system that is ideal for software development teams using monorepos that contain many components and dependencies. Zim provides for fast incremental, parallel builds across a team and is entirely language agnostic with built-in support for cross-platform builds via Docker. Zim is available as an open source project hosted on GitHub. 

In developing Zim, we drew inspiration from the core concepts of GNU Make and the caching strategy employed by Buck, Bazel, and Please. Like Make, Zim has a lightweight way to express new rules that define inputs, outputs, and the commands needed to create the outputs. Zim computes Rule Keys which are used to determine whether the output of a rule is already available in the cache, based on the combined hashes of the rule's inputs and configuration.

When Fugue began building our cloud security and compliance SaaS on Amazon Web Services (AWS), we favored a serverless approach via AWS Lambda and AWS Fargate as a way to avoid managing virtual servers (e.g., EC2 instances), gain security and isolation for jobs via an AWS mechanism, and allow us to scale in a very responsive and dynamic way. We opted to store all our code in  a monorepo and we scripted all build and deploy tasks using Makefiles. Leveraging CircleCI was a big win for us early on, since we could run unit tests, builds, and deploys with a great deal of parallelization.

However, as the system grew to hundreds of components, the Makefile approach grew unwieldy and we knew we needed improved tooling for builds and artifact management. A key aspect to introducing a new build tool was ease of migration from Makefiles. It was this requirement in particular that led us to prototype and then adopt Zim, rather than going with one of the existing open-source caching build tools. We needed the ability to quickly add new rules and output types, with simplicity akin to adding a new build target in a Makefile.

With Zim and its shared cache, we reduced build times for a number of common tasks by an order of magnitude. Reusing shared Zim build templates also significantly reduced ongoing maintenance of Makefile based build scripts. Our library of templates supports efficiently building Lambdas and binaries for Fargate that are written in Python, Go, and Node JS.

But, enough talk. Show me the code!

Here is an example Zim build template for compiling Go binaries for Linux. The build is cross-platform in that it can run on MacOS, Linux, or Windows (via WSL) to target the Linux platform. The output in this case is a zip file containing the Go binary. This would be suitable for use with AWS Lambda, for example, which accepts zip files that are unpacked by the Lambda runtime.

kind: go
docker:
 image: golang:1.15.0
toolchain:
 items:
   - name: go architecture and version
     command: go version
environment:
 GO_OPTS: -mod=readonly
rules:
 build:
   inputs:
     - go.mod
     - go.sum
     - "**/*.go"
   ignore:
     - "**/*_test.go"
   outputs:
     - ${NAME}.zip
   commands:
     - cleandir: build
     - run: go build ${GO_OPTS} -o build/${NAME}
     - zip:
         cd: build
         output: ../${OUTPUT}

This YAML build template is completely reusable across multiple components in the monorepo. Any number of components may declare themselves to be of “kind: go” and they will automatically use this build template. When a component needs a slight customization, each section of the template may be overridden in that component’s template.

You can then execute the following command to run a build. This automatically reads and writes from the team’s shared cache, so if the zip output file corresponding to your current inputs is already in the cache, Zim will simply download it for you instead of building it.

$ zim run build

Without additional options, zim run build will build all components in your monorepo. Here is an example that builds a single component:

$ zim run build -c my_component

rule: my_component.build
cmd: [cleandir] build
cmd: go build ${GO_OPTS} -o build/${NAME}
go: downloading github.com/aws/aws-sdk-go v1.25.3
...
go: downloading golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6
cmd: [zip] created ../${OUTPUT}
rule: my_component.build in 12.800 sec [OK]

Now that the artifact has been built, anyone else on the team that builds the same component -- with the same input files -- will instead download the cached artifact:

$ zim run build -c my_component

rule: my_component.build
rule: my_component.build [CACHED]

With Zim you can easily query the information used to build rule keys. This aids writing correct templates and provides visibility into all the inputs Zim uses to determine whether a new build should happen, or whether a cached artifact is already available. Use the following command to print a JSON object that represents the rule key:

$ zim key -r my_component.build --detail

Thanks for taking a few minutes to learn about Zim! We’ve found that Zim has the following advantages for our team:

• Fast, parallel builds. Rules run only if inputs have changed and outputs are pulled from a shared cache if someone else built it already.
• Trivial build step definitions. Define how to build new component types in a few lines of YAML.
• Isolated build environments and cross-platform compilation via the built-in Docker support. Just specify the Docker image to be used when building a component.
• Flexible input and output resource types. Zim is able to work with both files and Docker images as natively supported resources.
• Easy setup for a shared cache in S3 via an AWS CloudFormation stack.
• Lightweight & easy to install. Zim is written in Go and consists of a single binary when built. It is compatible with virtually any CI/CD pipeline.

Please give it a try and let us know what you think!

Zim is available on GitHub here.