Learn Rust Series (#56) - Cargo Workspaces and Multi-Crate Projects

scipio(71)

Published in

StemSocial

Words

3162

Reading

15 min

Listen

Play

Learn Rust Series (#56) - Cargo Workspaces and Multi-Crate Projects

What will I learn

You will learn what a Cargo workspace is and why large projects split into many crates;
how a workspace shares one Cargo.lock, one target/ directory, and one dependency graph;
how a binary crate depends on a library crate by path, and how they build together;
how to design a clean public surface with modules, pub use re-exports, and pub(crate);
when to split a common crate out so a client and a server share the same types.

Requirements

A working modern computer running macOS, Windows or Ubuntu, with Cargo;
An installed Rust toolchain (via rustup, from rustup.rs);
The previous fifty-five episodes, especially modules, crates, and visibility;
The ambition to learn systems programming from the ground up.

Difficulty

Intermediate

Curriculum (of the `Learn Rust Series`):

Learn Rust Series (#56) - Cargo Workspaces and Multi-Crate Projects

For fifty-five episodes we have quietly been living in a single crate. One Cargo.toml, one src/, one cargo build, and life was simple. But something has been accumulating under our feet: by now our projects have unit tests, integration tests, benchmarks, error types, and enough moving parts that "just put it all in lib.rs" is starting to creak. Real programs are almost never one crate. They are a library that holds the logic, a binary that wraps it in a command-line interface, maybe a second binary that runs a server, a set of benchmarks, and a shared crate of types that both the client and the server agree on. The tool Cargo gives you for exactly this is the workspace -- a way to make several crates live and build as one, sharing a single lockfile and a single build directory.

This episode is about structuring a real multi-crate project the way production Rust codebases actually do it. We will build up the mental model piece by piece -- the workspace manifest, a library crate, a binary that depends on it, a shared-types crate -- and then look at the visibility tools (pub use, pub(crate)) that let you present a clean public face while keeping your internals free to change. None of this is hard, but getting the shape right early saves you a painful reorganisation later, when half a dozen crates already depend on the layout you regret ;-)

Solutions to Episode 55 Exercises

Episode 55 was benchmarking, and all three exercises were about measuring honestly rather than guessing. Here they are in full.

Exercise 1 asked us to time sort versus sort_unstable on the same large vector of random u64s, averaging over many runs, remembering to black_box and to re-clone before each sort (because sorting mutates in place, so a second sort on an already-sorted slice measures nothing):

use std::hint::black_box;
use std::time::{Duration, Instant};

fn make_data(n: usize) -> Vec<u64> {
    let mut x = 0x9E3779B97F4A7C15u64; // any odd seed
    (0..n).map(|_| { x = x.wrapping_mul(6364136223846793005).wrapping_add(1); x }).collect()
}

fn avg_sort(base: &[u64], runs: u32, unstable: bool) -> Duration {
    let start = Instant::now();
    for _ in 0..runs {
        let mut v = base.to_vec();          // fresh unsorted copy every run
        if unstable { v.sort_unstable(); } else { v.sort(); }
        black_box(&v);                      // stop the optimizer deleting the sort
    }
    start.elapsed() / runs
}

fn main() {
    let base = make_data(100_000);
    println!("stable:   {:?}", avg_sort(&base, 50, false));
    println!("unstable: {:?}", avg_sort(&base, 50, true)); // usually a touch faster
}

The re-clone inside the loop is the whole trick: sort the same Vec twice and the second call sees a sorted slice and returns almost instantly, so you would be timing "detect already-sorted" rather than "sort". sort_unstable is typically a little quicker because it does not have to preserve the relative order of equal elements, and for a vector of plain u64s (where equal means identical) you lose nothing by using it.

Exercise 2 wanted a benchmark body run once with black_box around the result and once without, reporting both and explaining why the version without it is suspiciously fast:

use std::hint::black_box;
use std::time::Instant;

fn work(n: u64) -> u64 {
    (1..=n).map(|x| x.wrapping_mul(x)).sum()
}

fn main() {
    let start = Instant::now();
    for _ in 0..1000 { let _ = work(black_box(50_000)); }   // result observed via black_box
    println!("with    black_box: {:?}", start.elapsed() / 1000);

    let start = Instant::now();
    for _ in 0..1000 { let _ = work(50_000); }              // result discarded, input a constant
    println!("without black_box: {:?}", start.elapsed() / 1000);
}

The version without black_box is suspiciously fast because the input is a compile-time constant and the result is never observed, so the optimizer is free to compute work(50_000) once (or fold it away entirely) and skip the other 999 iterations -- you end up timing an empty loop.

Exercise 3 asked us to measure a function across four input sizes, state from the numbers whether the growth is linear or quadratic, then compute throughput in MiB/s for the largest size:

use std::hint::black_box;
use std::time::Instant;

fn work(n: u64) -> u64 {
    (0..n).map(|x| x & 0xff).sum()
}

fn main() {
    for &size in &[1_000u64, 10_000, 100_000, 1_000_000] {
        let start = Instant::now();
        black_box(work(size));
        let secs = start.elapsed().as_secs_f64();
        if size == 1_000_000 {
            let mib = size as f64 / (1024.0 * 1024.0); // one byte processed per element
            println!("n={size:>9} took {secs:.6}s = {:.1} MiB/s", mib / secs);
        } else {
            println!("n={size:>9} took {secs:.6}s", start.elapsed().as_secs_f64());
        }
    }
}

If ten times the input costs roughly ten times the time, the growth is linear -- which is what this sum will show. Quadratic growth would multiply the time by a hundred each step, and spotting that difference in a column of numbers is exactly how you catch an accidental O(n^2) before it reaches production. Right, homework cleared. Now, workspaces.

The workspace manifest

A workspace is a top-level Cargo.toml that groups a set of member crates. The important and slightly surprising part is that this top-level manifest usually has no [package] section of its own -- it is a virtual manifest whose only job is to list the members. Those members then share one Cargo.lock and one target/ directory between them:

# top-level Cargo.toml (a virtual manifest -- no [package] of its own)
[workspace]
resolver = "2"
members = ["core", "cli", "common"]

Running cargo build at the root builds every member in dependency order; cargo build -p cli builds just the one you name; cargo test at the root runs every crate's tests in one go. The single shared Cargo.lock is the quiet hero here: it guarantees that every member resolves each third-party dependency to the same version. That is what prevents the maddening "expected serde::Value, found serde::Value" errors you get when two crates accidentally pull in two different versions of the same library and the compiler treats their types as unrelated. In a workspace, there is one graph and one answer.

The resolver = "2" line is worth a word. It selects Cargo's second-generation feature resolver, which handles feature unification more sanely across a dependency graph (it does not force a dev-dependency's features onto your normal build, for instance). It is the default for the 2021 edition and later, but in a virtual workspace manifest you often have to state it explicitly because there is no [package] edition for Cargo to infer it from.

Sharing versions and metadata across members

Once you have three or four crates, repeating the same serde = "1" line and the same version, edition, and license in every member's manifest gets old fast. Cargo lets the workspace root declare shared values that members inherit with workspace = true:

# top-level Cargo.toml
[workspace.package]
version = "0.1.0"
edition = "2021"
license = "MIT"

[workspace.dependencies]
serde = { version = "1", features = ["derive"] }

# core/Cargo.toml
[package]
name = "core"
version.workspace = true      # inherit 0.1.0 from the root
edition.workspace = true

[dependencies]
serde.workspace = true         # inherit the exact serde spec from the root

Now there is a single place to bump the shared serde version or the workspace-wide edition, and every member follows. This is not just tidiness -- it is how you keep a growing workspace from drifting into a mess of slightly-different version specs that the resolver then has to reconcile.

A library crate holds the logic

With the manifest understood, let us look at the members themselves. The core crate is a plain library: a src/lib.rs exposing public functions and types, and knowing nothing about the command line, the network, or how it will be used. That ignorance is a feature -- it is precisely what makes the crate reusable and trivial to test in isolation:

// core/src/lib.rs
pub fn area(width: f64, height: f64) -> f64 {
    width * height
}

pub fn perimeter(width: f64, height: f64) -> f64 {
    2.0 * (width + height)
}

Keeping the logic in a library and out of main.rs is one of those habits that pays for itself almost immediately. A function in a library crate can be unit-tested, benchmarked (as we saw last episode), documented with doctests (episode 52), and reused by a second binary -- none of which is comfortable when the same code is buried inside a fn main. As a rule of thumb: main.rs should be thin, just argument parsing and calls into the library, while lib.rs carries the weight.

A binary crate depends on it by path

The cli crate is the thin wrapper. It lists core as a path dependency in its own manifest and then imports from it exactly as if it were any crate from the registry. Because I want the example to compile in a single file here, I will model the separate crate with a module standing in for it -- but the comment shows the real relationship:

// cli/src/main.rs -- in reality `use core::area;` where core is a path dependency
mod core {
    pub fn area(w: f64, h: f64) -> f64 { w * h }
}

fn main() {
    let (w, h) = (3.0, 4.0);
    println!("area = {}", core::area(w, h)); // area = 12
}

The real cli/Cargo.toml would carry a single line under [dependencies]:

# cli/Cargo.toml
[dependencies]
core = { path = "../core" }

and Cargo, seeing that cli depends on core, compiles core first and hands its compiled artifact to cli. Path dependencies are the glue of a workspace: they wire the members together locally, with no registry involved, and Cargo tracks them so that changing core triggers a rebuild of everything downstream of it. Nota bene: a member can even carry both a path and a version (core = { path = "../core", version = "0.1" }), so it uses the local copy during development but resolves against the published version once it is on crates.io -- handy for libraries you intend to release.

A shared types crate

Here is the scenario that makes workspaces click. Suppose you have two binaries -- a server and a client -- that exchange messages. If each defines its own Message struct, you have two definitions that can silently drift apart, and the compiler cannot help you because it has no idea they are meant to be the same. The fix is a common crate holding the shared shape, which both binaries depend on. Now there is exactly one definition, and the compiler enforces that both sides speak it identically:

// common/src/lib.rs
#[derive(Debug, Clone, PartialEq)]
pub struct Point {
    pub x: i32,
    pub y: i32,
}

impl Point {
    pub fn origin() -> Self {
        Point { x: 0, y: 0 }
    }
}

Both the client and the server put common = { path = "../common" } in their manifests, and from that moment they cannot disagree about what a Point is -- change a field in common and both sides stop compiling until they are updated together. That is the whole value proposition: a shared crate turns "two things that ought to match" into "one thing that cannot help but match". This pattern is everywhere in real Rust -- shared protocol types, shared configuration structs, shared error enums.

Designing the public surface with pub use

Inside a crate you will naturally organise code into modules, but you almost never want to force your users to know that internal layout. A facade re-exports the important items at the crate root with pub use, so callers import from one tidy path instead of spelunking through your module tree:

mod shapes {
    pub fn circle_area(r: f64) -> f64 { std::f64::consts::PI * r * r }
}
mod solids {
    pub fn sphere_volume(r: f64) -> f64 { 4.0 / 3.0 * std::f64::consts::PI * r * r * r }
}

pub use shapes::circle_area;   // re-exported at the crate root
pub use solids::sphere_volume;

fn main() {
    println!("{:.3}", circle_area(1.0));   // 3.142
    println!("{:.3}", sphere_volume(1.0)); // 4.189
}

Your users write use mycrate::circle_area and never once mention shapes or solids. The payoff is that those internal module names become yours to change -- rename shapes, split it in two, merge it into solids -- and as long as the re-exports at the root still point somewhere valid, not a single downstream crate breaks. This is exactly how the big crates you already use are built: std itself re-exports a great deal at convenient paths rather than exposing its raw module structure. The public API is a promise; pub use lets you keep that promise while leaving everything behind it free to move.

pub(crate): visible internally, hidden externally

Between fully private (a plain fn, visible only in its own module) and fully public (pub, visible to the whole world) sits a middle ground that workspaces make you appreciate: pub(crate). An item marked pub(crate) is reachable anywhere inside your crate, but completely invisible to any crate that depends on you. It is the perfect visibility for helpers that several of your modules share but that have no business being part of your public API:

mod engine {
    pub(crate) fn internal_tick() -> u64 { 42 } // shared across the crate, not public API
    pub fn public_run() -> u64 { internal_tick() * 2 }
}

fn main() {
    println!("{}", engine::public_run());   // 84
    println!("{}", engine::internal_tick()); // reachable here -- same crate
}

From a dependent crate, engine::internal_tick would be a compile error, which is exactly the boundary you want: your modules can freely share it, but it never leaks into the surface that others depend on. Rust actually gives you a whole ladder of visibility here -- pub(self) (the default, private), pub(super) (visible to the parent module), pub(crate) (whole crate), pub(in some::path) (a named ancestor), and finally pub (everyone). In practice pub(crate) is the one you will reach for constantly once a crate grows past a couple of modules, because it lets you refactor internals fearlessly: nothing outside can possibly depend on what nobody outside can see.

Testing across the workspace

Testing scales up naturally. Each crate keeps its own unit tests in a #[cfg(test)] module (episode 52), and the workspace as a whole can hold integration tests that exercise several crates together through their public APIs. cargo test run at the root walks every member and runs the lot:

pub fn add(a: i32, b: i32) -> i32 { a + b }

#[cfg(test)]
mod integration {
    use super::*;
    #[test]
    fn adds() {
        assert_eq!(add(2, 2), 4);
    }
}

The nice property is that the same commands you already know -- cargo test, cargo build, cargo doc -- all become workspace-aware for free. cargo test -p common tests just one member; cargo test --workspace is explicit about testing everything; cargo doc --workspace --no-deps builds one unified documentation site spanning all your crates, with links between them working out of the box. Your whole toolchain simply understands the workspace as a unit.

How Go, Python and C approach it

A glance sideways always sharpens the picture. Go organises code into packages inside a module, and for the multi-module case it added go.work files -- a workspace concept that is a near-cousin of Cargo's, letting several modules resolve against each other locally during development:

// go.work -- lets the cli module use a local copy of the common module
go 1.21

use (
    ./core
    ./cli
    ./common
)

Go's visibility, though, is cruder than Rust's: an identifier is exported if it starts with a capital letter and private otherwise, full stop. There is no pub(crate) middle ground -- if you want something shared across packages but not truly public, the language does not help you, and you lean on the internal/ directory convention (packages under internal/ are importable only by their parent tree). Python has no compiler and no real notion of a workspace; a project is a package (a directory with an __init__.py, or these days a pyproject.toml), and "multi-crate" is approximated with either multiple installable packages or a monorepo of them wired together with editable installs (pip install -e). Visibility is by convention only -- a leading underscore says "please do not touch this", and nothing enforces it. C and C++ have no built-in package system at all; the moral equivalent of a workspace is a build system like CMake gluing together several libraries and executables, with visibility managed by which symbols you expose in header files versus keep static in a .c file. That the C world manages this entirely through discipline and tooling, with no language-level help, is a good reminder of how much Cargo is quietly doing for you -- one command builds the graph, one lockfile pins the versions, and the module system enforces the boundaries that C leaves to header hygiene.

Wrapping up

So here is the shape of it. A workspace is a virtual top-level Cargo.toml that lists member crates, which then share one Cargo.lock and one target/, so every crate resolves dependencies to the same versions and cargo build/test/doc at the root operate on the whole thing at once. Put your logic in a library crate so it can be tested, benchmarked, and reused; keep binaries thin wrappers that depend on it by path; and when two crates must agree on a data shape, give that shape a home in a shared common crate so the compiler enforces the agreement. Present a clean face with pub use re-exports at the crate root, and keep your internals honest with pub(crate) so nothing outside can depend on what it should not see. And -- Having said that -- resist the urge to split too early: start with one crate, break out a library when the logic earns independent testing, and add a common crate only when two binaries genuinely share types. A workspace is a tool for managing real structure, not a checklist to tick off on day one.

We now have projects with several crates, tests, benches, and a proper public surface. The next thing that structure invites is variation: the same crate needing to build slightly differently for different situations -- with an optional feature switched on here, for a different target platform there, compiling one block of code and skipping another. Rust handles all of that at compile time rather than at runtime, and next time we look at the machinery that lets a single codebase produce many tailored builds ;-)

Exercises

Sketch a two-crate workspace with a core library and a cli binary: write area in core and call it from cli. Use a module stand-in (as shown above) so it compiles in one file, and add a comment showing the real path = "../core" dependency line.
Re-export two functions that live in separate internal modules through a facade with pub use, then call both from the crate root as if the internal modules did not exist. Rename one internal module and confirm the public call still works.
Write a common-style struct deriving Debug, Clone, and PartialEq, mark one internal helper pub(crate), and confirm a pub wrapper that calls the helper still compiles while a direct call to the helper would be the crate's private business.