BazelCon 2024

In this session I will discuss remote action deduplication, a feature of some Bazel RBE backends. I will explain the problem it solves, how it works, and some details about how well it has worked for us. I'll also quickly touch on merge queues, and mention some RBE backends that support the feature. --- If you run builds on every master commit, and you have commits landing more quickly than your slowest action can finish, you'll end up rebuilding unnecessarily. We estimated that our slowest webpack action was being re-run unnecessarily 66% of the time! This action is an input to a large number of expensive integration tests so it's very likely that we're also wasting a lot of compute on those tests as well. The web page builds and tests account for the vast majority of the compute we use in CI. The number of wasted builds would be even higher if we were using a merge queue, so we have been forced to stick to a conventional merge strategy. By running these builds remotely and using an RBE backend that supports action deduplication, we've massively (25%, more expected by BazelCon) reduced the amount of compute needed to build and test web pages.

Container images are a ubiquitous mechanism for deploying applications in the modern computing landscape. Everything from services deployed on bare-metal Kubernetes to AWS Lambda functions depend on container images. Building these images quickly and efficiently is critical to keeping CI/CD responsive. In recent years, Stripe has undergone dual migrations. One, to a service oriented architecture. Two, to running services inside Kubernetes. Together, these migrations meant our CI pipelines to build containers became slower and more expensive with each passing month. With a build producing over a terabyte of images, Bazel just couldn't build them fast enough! Even disk space was in short supply. Learn how we flipped the script on our container builds by leveraging BwoB (Build without the Bytes). Now, we can build 1300+ container images in just a couple minutes with plenty of headroom for more. Bazel's entire output space fits in RAM and there's never a single byte of container images on disk!

In this talk, you'll gain practical insights from our experience migrating Airbnb's web monorepo (which houses millions of lines of TS/JS code) to Bazel. We developed methods and tooling to trim down input sizes, share the cache between MacOS and Linux, persist caches through local and remote builds, catch and fix non-deterministic builds, and automatically generate BUILD files from JS/TS code. This helped us create a robust system for adopting Bazel without compromising on developer experience. These insights are designed to help you navigate similar migrations, improve build performance, and reuse builds across platforms.

Many at this conference are familiar with Remote Caching and Remote Execution. Over at BuildBuddy, we’re excited to introduce Remote Bazel - running the Bazel client server and command remotely as well, making your builds even faster and more correct. Even if your builds are triggered by ephemeral CI runners, Remote Bazel will guarantee your builds always run with a warm analysis and repository cache, and remove the need to pull images, install dependencies, and clone git repos on each run. Running builds in the same datacenter as the remote cache and executors also reduces network bottlenecks and egress costs. Even if your company’s developers use machines with different operating systems or versions of tools or don’t even have Bazel installed, Remote Bazel will guarantee that important builds are run in the exact same environment every time. This talk will provide a quick overview of how we built this and the many applications we’re excited about - from speeding up CI to augmenting our UI to improving the daily developer experience via the CLI.

Learn how to get more value from Bazel for your developer platform with faster migrations and consistently fast builds and tests around the world with cache replication, seamlessly scale remote build execution from 100 to 100,000 cores, get proactive recommendations on further performance improvements, and achieve cloud cost savings from thousands to millions of dollars.

Aspect authors many of the Bazel rules you know and love. We also provide the missing CI/CD and developer platform to integrate Bazel into your team's workflows and achieve the promised incremental build&test speed while reducing your cloud spend! This talk will update you on our latest and greatest.

It took me a while to understand how cc_shared_library works in Bazel. After fighting the system and having reached every possible error message, I took a deep dive into the underlying logic and discovered how powerful that rule is once you understand its opinionated view on shared libraries and the mindset behind the design of the rule. I was also surprised to discover that it relies on Bazel aspects. To the best of my knowledge it is the only core rule that does so. In this presentation I will convey my insights using diagrams depicting how one should reason about cc_shared_library. This will cover how shared libraries are assembled from cc_libraries, how they interact as dependencies of other cc_* rules, and some interesting aspects of the implementation based on… aspects!

Actually Portable Executables (APE) is a format of executable that can be ran on Windows, Linux, MacOS, BSDs operating systems from a single binary. A binary can be "fat" to contain both the x86_64 and aarch64 machine code to enable execution on multple CPUs from the same single binary. The `@ape` Bazel module provides the necessary wrappers and launchers to provide common POSIX tooling across all operating systems and CPUs such as `cp`, `mv`, `sed`, `awk` and `curl`. This talk will provide usage information of the `@ape` tools to enable an ecosystem of hermetic tooling for Bazel rulesets that require zero system setup to be used other than a working Bazel installation.

The talk will be split into a few sections:

- An introduction to the Bazel module, it's usage and benefits
- A high level overview into the APE format, Cosmopolitan C library and superconfigure to provide attendees relevant information to understand what the module is providing
- A brief introduction to the ecosystem of hermetic rulesets that expose rules around the APE hermetic binaries, such as `rules_curl`, `rules_gzip`, etc.

Bazel’s rules_python supports using a hermetic python environment at build and test time. However, maintaining sufficient similarity between the environment that unit tests are running in under bazel test and the production environment can be challenging, particularly once things like extension modules with nontrivial native dependencies get involved. The conda ecosystem is primarily for python packages, but also includes packages for nearly all of the native dependencies of those python packages as well. This talk describes the bazel rule-set we use to fetch conda packages and deploy them for use both during the bazel build and test, and in production. These rules allow us to provide our customers with a deployment package which they can simply untar and run on a wide range of linux environments with very limited prerequisites, producing several distinct (but overlapping) environments for different products within our monorepo, at the same time ensuring that our unit tests are running in a nearly identical environment, all while maintaining acceptable build times.

While Bazel excels in speed and accuracy, its integration with native ecosystems, particularly IDE support, has been lacking. Jay Conrod’s 2022 BazelCon talk highlighted how this hinders adoption, echoing our observations at LinkedIn. The model recommended in that talk was to have the language server use a build adapter as a liaison to Bazel to correctly resolve imports. Since then, Bzlmod and Uber’s go_deps module extension have enabled us to treat go.mod as the source of truth for Go dependencies thereby removing the need for lengthy WORKSPACE files and constant updates via gazelle-update-repos. While these significantly moved the needle, they didn’t support multi module workspaces, still relied on the build adapter model for IDE support and had no native CLI support. At LinkedIn, we sought to address these gaps. We added Go workspace support to Gazelle and introduced a setup rule wherein Bazel bootstraps the Go SDK, sets Go env vars such as GOPATH within the Bazel workspace and informs IDEs and shells via settings files and direnv, thus providing a seamless developer experience. This has enabled us to drive developer adoption across a poly repo ecosystem at LinkedIn.

In a repo with almost a million files, `gazelle` historically scales O(size of repo) for automatic BUILD file generation, since gazelle plugins need to read and evaluate source files across the repo in order to prepare the BUILD files' contents.

When the peak runtime of gazelle approached 5-8 minutes, this became unscalable at Uber, with developers being blocked and delayed. Our team was able to improve three key metrics related to Gazelle's speed:
- Gazelle p75 on CI from >300s to <10s
- Gazelle being part of critical path CI 40% to <5%
- Local developer gazelle p95 from 30s to <10s

The team was able to achieve these results through these technical enhancements:
- Create a framework to make `gazelle` run incrementally at O(size of change) rather than O(size of repo) for almost all changes.
- Profile latency blockers and non-performant plugins to improve the performance of the gazelle binary itself.
- Pre-build and package the `gazelle` binary and its plugins with Bazel to avoid the overhead of building the binary.

This session will focus on how Uber integrated with Gazelle for scalability, extensibility, and performance in our huge Monorepo.