Rationale

A perennial topic of interest to software engineers is our tooling. We are usually on the lookout for better, faster, more reliable tooling. In the case of Housecarl AuthZ, I went a little off the beaten path by building a web application.

First, Rust is chosen as the backend programming language for Housecarl. It is the standard, as it were. Time and effort are invested here.

Go has been chosen as the scripting language - it also serves as the server to route the static web content from Google Cloud Storage (S3 in Google colors) to your computer.

Go is not a conventional scripting language, but it provides two essential benefits:

1. Speed of compilation. go run script.go works very fast and will run very fast.
2. Static typing. One of the most perennial headaches for *nix users is Bash scripts grown sprawling and interpolation bugs creeping in. By selecting a statically typed conventional programming language, the sprawl can be managed with refactors that Bash inhibits by its nature.

The current front end of the web UI is done with elementary Javascript without frameworks or libraries such as React or Vue. This is for two reasons:

1. The "meat" of Housecarl is in Rust. Attending excessively to a UI detracts from time better spent in making the backend excellent.
2. Simplicity of development. The Javascript is modern and well documented. Without attending to build systems or complex frameworks, it "gets out of the way". Naturally, if we spend a good deal of time working on the UI, a homegrown framework will result. But for now, this is acceptable.

In the future, it is likely that the front end of the web UI will be redone in React or similar. Some of the complexity is significant.

These are all very different systems and can induce fairly large headaches if tied together in either a single repository with a number of shell scripts, or tied together in a number of repositories with, well, shell scripts and some duct tape.

The solution I came to, after many years of work, is to use a single repository - it ties together the system in a consistent and predictable state - and then to use a build tool geared towards monorepos, Bazel.

I looked at a few other tools - Buck2, Pants, Please, but Bazel was the original open source edition, and the others are essentially Bazel with tweaks. Bazel historically was used at Google and known as Blaze. Buck2 is rewritten in Rust but my explorations indicated it wasn't mature enough for use.

Bazel does several things for this project:

• Rust compilation is properly cached. cargo uses the mtime feature, which generally is fine, but if you're switching branches, you have vendor'd code, you wind up doing rebuilds far too much.

• I can correctly tie together a Rust build and a Go build. For instance, it turns out that Go has a better SDK for integration with Stripe than Rust does. So I can write the Stripe integration in Go, and then roll it out simultanously with the Rust code, ensuring that the versioning is reliable.

• I can farm out builds from my local machine into remote build servers trivially - and the remote caching as well. I found that Buildbuddy works fantastically. My $200 Chromebook is now able to tidly build Rust at really a decent speed.

Details

I've written up a minimal Rust repository at staff. The essential goal I am demonstrating there is performing a Rust build with Bazel. Bazel serves as an agnostic system for performing the build.

Notice we need essentially 3 files:

• BUILD
• WORKSPACE.bzlmod
• MODULE.bazel

This is very simple, generating a single binary.

More sophisticated builds will obviously do far more.

One of the approaches some use in Rust for Protobufs is to have them generated at build time. I've elected to move that to a pre-compile time, in order to simplify development as well as to not have "magic" source code generated that will potentially confuse IDEs and audits.

The process roughly looks like follows:

• Write the .protos and check them in.
• Compilation of the .protos, targeting the relevant language.
  • Note that as quality pass, the protoc compilation should occur on each MR which has a protoc change, and if any change at MR-time is proposed, the dev failed to check in the protoC generated code correctly - fail.
• Check in the generated code.
• Integrate the generated code with the system.