Learn Rust Series (#52) - Testing: Unit Tests, Integration Tests, and Doctests

scipio(71)

Published in

StemSocial

Words

3231

Reading

15 min

Listen

Play

Learn Rust Series (#52) - Testing: Unit Tests, Integration Tests, and Doctests

What will I learn

You will learn how Rust builds testing into the language and cargo, with no external framework to pick or configure;
how to write unit tests in a #[cfg(test)] module right next to the code they test;
the assert!, assert_eq!, assert_ne! and #[should_panic] tools, plus tests that return Result;
what integration tests in the tests/ directory are, and why they see only your public API;
what doctests are, and why the examples in your documentation are compiled and run for you.

Requirements

A working modern computer running macOS, Windows or Ubuntu, with Cargo;
An installed Rust toolchain (via rustup, from rustup.rs);
The previous fifty-one episodes, especially modules (episode 9), Drop and RAII (episode 20), Result (episode 6 and 46-48), and panics (episode 51);
The ambition to learn systems programming from the ground up.

Difficulty

Intermediate

Curriculum (of the `Learn Rust Series`):

Learn Rust Series (#52) - Testing: Unit Tests, Integration Tests, and Doctests

One of the quiet joys of Rust is that testing is not a bolted-on library you have to shop around for, argue about, and configure. It ships in the language and in cargo. You write #[test] functions, you run cargo test, and a built-in harness compiles them, runs them in parallel, and reports the results with green ok lines and red failures. Coming from Python, where you pick between unittest, pytest, nose and a small graveyard of abandoned runners, this is a genuine relief -- there is one blessed way, everybody uses it, and it is already installed.

On top of that Rust does something most languages simply do not: it compiles and runs the code examples in your documentation. Your docs cannot silently rot into lies, because if an example stops compiling or its assertion fails, your test suite goes red and you find out. Last episode we closed by saying we would start putting our code under a microscope -- proving it behaves, including the panics that are supposed to happen. This is that episode. Let us walk through all three layers of Rust testing, and prove every claim with code you can run ;-)

Solutions to Episode 51 Exercises

Episode 51 was panics, unwinding, and containment. Here are all three exercises worked out in full.

Exercise 1 asked for a function that returns a Result in stead of panicking on bad input, called with a match on both arms -- and then a second version using expect, with a message documenting the invariant that makes the expect safe:

// version A: return a Result and handle BOTH arms explicitly
fn parse_age(s: &str) -> Result<u8, String> {
    s.trim()
        .parse::<u8>()
        .map_err(|e| format!("'{s}' is not a valid age: {e}"))
}

fn main() {
    match parse_age("34") {
        Ok(age) => println!("age is {age}"),      // age is 34
        Err(msg) => println!("rejected: {msg}"),
    }
    match parse_age("oops") {
        Ok(age) => println!("age is {age}"),
        Err(msg) => println!("rejected: {msg}"),  // rejected: 'oops' is not a valid age: ...
    }

    // version B: an invariant WE control, so expect is legitimate here
    let literal = "255";
    let max = literal
        .parse::<u8>()
        .expect("the literal \"255\" always fits in u8"); // documented invariant
    println!("max is {max}"); // max is 255
}

The distinction is the whole lesson: parse_age takes external input, so it hands back a Result and the caller decides. The literal "255" is a constant we wrote ourselves, so an expect whose message states why it can never fail is honest, not lazy.

Exercise 2 wanted catch_unwind wrapped around a closure that indexes past the end of a slice, printing whether it panicked, then a downcast_ref to recover the payload as text:

use std::panic;

fn main() {
    let data = [1, 2, 3];
    let outcome = panic::catch_unwind(|| data[10]); // index out of bounds
    println!("panicked? {}", outcome.is_err()); // panicked? true

    if let Err(payload) = outcome {
        // a slice-index panic carries a formatted String payload;
        // a panic!("literal") carries a &str -- so we try both
        if let Some(s) = payload.downcast_ref::<String>() {
            println!("message: {s}");
        } else if let Some(s) = payload.downcast_ref::<&str>() {
            println!("message: {s}");
        } else {
            println!("non-string payload");
        }
    }
}

Note the sharp edge worth remembering: an out-of-bounds panic builds its message with format!, so the payload is a String, whereas panic!("literal text") stores a &'static str. That is why we try both downcasts -- the payload type depends on how the panic was raised.

Exercise 3 asked you to spawn three threads where exactly one panics, join all three, and count survivors versus casualties -- proving the two healthy threads were untouched:

use std::thread;

fn main() {
    let handles: Vec<_> = (0..3)
        .map(|id| thread::spawn(move || {
            if id == 1 {
                panic!("worker {id} hit an unrecoverable state");
            }
            id * 10 // survivors return a value
        }))
        .collect();

    let (mut finished, mut panicked) = (0, 0);
    for h in handles {
        match h.join() {
            Ok(value) => { finished += 1; println!("worker returned {value}"); }
            Err(_) => { panicked += 1; }
        }
    }
    println!("{finished} finished, {panicked} panicked"); // 2 finished, 1 panicked
}

The panic never crosses the thread boundary -- it is captured in the Err arm of join, the two survivors return their values through the Ok arm, and main tallies the score. That containment pattern is exactly what real thread pools are built on. Right, homework cleared. Now, testing.

A unit test lives next to its code

The idiomatic place for a unit test is a module in the same file as the code it exercises, gated with #[cfg(test)]. That attribute is the key: cfg(test) is only true when you compile with cargo test, so the entire module -- tests, helpers, imports and all -- is stripped out of your shipped binary. You pay nothing in your release build for a wall of tests. Each test is an ordinary function marked #[test] that panics to signal failure, usually through an assertion macro:

pub fn add(a: i32, b: i32) -> i32 {
    a + b
}

#[cfg(test)]
mod tests {
    use super::*; // bring the parent module's items into scope

    #[test]
    fn adds_two_positive_numbers() {
        assert_eq!(add(2, 3), 5);
    }

    #[test]
    fn adds_with_a_negative() {
        assert_eq!(add(5, -8), -3);
    }
}

That use super::*; is doing real work. A #[cfg(test)] mod tests is a child module of the file it lives in (remember the module tree from episode 9), and a child can reach into its parent -- including private items. This is the entire point of a unit test: it can poke at your internal helpers, your private fields, the little functions you never exported, not just the public surface. When you want to test the machinery, not the facade, this is where you do it.

Run cargo test and the harness collects every #[test] function it can find, runs them, and prints a report:

running 2 tests
test tests::adds_with_a_negative ... ok
test tests::adds_two_positive_numbers ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Notice the order is not the source order -- tests run in parallel, on multiple threads, which is a detail with teeth we will come back to.

The assertion macros

Three macros cover almost everything you will ever write. assert! checks that a boolean is true. assert_eq! checks that two values are equal, and assert_ne! checks that they are not -- and both of those beat a bare assert! for equality, because on failure they print both sides for you. There is a world of difference between "assertion failed" and "left: 14, right: 15" when you are staring at a red test at the end of a long day:

pub fn is_prime(n: u32) -> bool {
    if n < 2 { return false; }
    (2..n).take_while(|d| d * d <= n).all(|d| n % d != 0)
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn small_primes() {
        assert!(is_prime(2));
        assert!(is_prime(13));
        assert_eq!(is_prime(1), false);
        assert_ne!(is_prime(15), true); // 15 = 3 * 5, not prime
    }
}

Every assertion macro also takes an optional trailing message with format arguments, which is worth adding whenever the failure would not be self-explanatory. Something like assert!(user.is_active, "user {} should be active after login", user.id) turns a bare panic into a sentence that tells future-you what went wrong without opening the source. Having said that, do not over-narrate the obvious -- a plain assert_eq!(add(2, 3), 5) already reports everything you need.

There is a companion macro I reach for constantly during development: dbg!. It prints the file, line, expression and value, then returns the value, so you can wrap it around any sub-expression to peek at it without restructuring your code. It is not a test tool as such, but it is your best friend while writing tests.

Testing that something panics

Some functions are supposed to panic on bad input -- and last episode we spent a lot of time on exactly when that is the right design. So how do you test a deliberate panic? You mark the test #[should_panic], and you make it precise by supplying expected, a substring that must appear in the panic message. Without expected, a test could pass on the wrong panic (a typo panicking for a different reason than you intended), which is a nasty way to be fooled:

pub fn checked_div(a: i32, b: i32) -> i32 {
    if b == 0 {
        panic!("division by zero is undefined");
    }
    a / b
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    #[should_panic(expected = "division by zero")]
    fn dividing_by_zero_panics() {
        checked_div(10, 0);
    }

    #[test]
    fn ordinary_division_works() {
        assert_eq!(checked_div(10, 2), 5);
    }
}

The expected string only needs to be a substring of the real message, so "division by zero" matches "division by zero is undefined" fine. This connects straight back to episode 51: a #[should_panic] test is how you assert that your invariant checks actually fire, that your panic! and your unwrap-on-a-true-invariant behave as documented.

Tests that return Result

A test does not have to return (). It can return Result<(), E>, and then a returned Ok(()) counts as a pass while an Err counts as a failure. The payoff is that you can use the ? operator inside the test, which reads far more naturally than a tower of unwraps when your test itself performs several fallible steps:

pub fn parse_pair(s: &str) -> Result<(i32, i32), std::num::ParseIntError> {
    let mut it = s.split(',');
    let a = it.next().unwrap_or("").trim().parse()?;
    let b = it.next().unwrap_or("").trim().parse()?;
    Ok((a, b))
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn parses_a_comma_pair() -> Result<(), std::num::ParseIntError> {
        let (a, b) = parse_pair("3, 4")?; // the ? bubbles a parse failure up as a test failure
        assert_eq!(a + b, 7);
        Ok(())
    }
}

One thing to keep in mind: a Result-returning test signals failure by returning Err, not by panicking, so you cannot combine it with #[should_panic]. Use the Result shape when the test does fallible setup and you want ?, and the plain shape when you are asserting a panic. Pick the tool that matches what you are actually checking.

Skipping the slow ones

Not every test should run on every cargo test. A test that hammers a large computation, or one that would need a network you do not always have, can be marked #[ignore] so it is skipped by default and only runs when you explicitly ask with cargo test -- --ignored:

pub fn triangular(n: u64) -> u64 {
    (1..=n).sum()
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn small_case_is_fast() {
        assert_eq!(triangular(10), 55);
    }

    #[test]
    #[ignore = "slow; run with: cargo test -- --ignored"]
    fn the_expensive_one() {
        assert_eq!(triangular(1_000_000), 500_000_500_000);
    }
}

That = "reason" on the attribute is optional but kind -- it documents why the test is ignored, so nobody wonders whether it was disabled because it is slow or because it is broken.

Integration tests see only your public API

Everything so far has been a unit test living inside your crate. Integration tests are a different animal. They live in a separate tests/ directory at the root of your project, alongside src/, and each file in there is compiled as its own separate crate that depends on yours -- exactly the way a real user's project would. The consequence is important: an integration test can only touch your public items. Private helpers are invisible to it, just as they are to anyone who adds your crate to their Cargo.toml.

That makes integration tests a brilliant check on whether your public API is actually usable. If something is awkward to call from tests/, it will be awkward for your users too, and you have found that out before you shipped. A file at tests/api.rs would look like this (I have inlined a stand-in for the library function so the snippet stands alone, but in a real project you would write use my_crate::add;):

// file: tests/api.rs -- compiled as its OWN crate that depends on yours
// In a real project the next line is how you reach the library:
//     use my_crate::add;

pub fn add(a: i32, b: i32) -> i32 { a + b } // stand-in for the imported function

#[test]
fn public_api_adds() {
    assert_eq!(add(40, 2), 42);
}

You do not write #[cfg(test)] inside tests/ files, by the way -- the whole directory is only ever compiled during cargo test, so the attribute would be redundant. If several integration test files need shared helpers, you put them in tests/common/mod.rs (the mod.rs form specifically, so cargo does not treat it as its own test crate) and pull them in with mod common;.

The split is deliberate and worth internalising: unit tests guard the internals, integration tests guard the contract. A healthy crate has both. The unit tests let you refactor private machinery with confidence; the integration tests let you evolve the public API without accidentally breaking your promise to users.

Doctests: your documentation is a test suite

Here is the feature that genuinely surprises people arriving from other languages. Any fenced code block placed inside a /// documentation comment becomes a doctest. When you run cargo test, it extracts each of those examples, wraps it in a tiny main, compiles it, and runs it. Your examples therefore cannot drift out of date -- the moment one stops compiling, or an assertion inside it fails, your test suite turns red and you go fix it.

I cannot show you the literal triple-backtick block inside a doc comment here without confusing this article's own code fences, so picture the doc comment written in the text block below, and then look at the rust block after it, which is the exact body cargo test would compile and run from that example:

/// Doubles a number.
///
/// # Examples
///
/// ```
/// let result = my_crate::double(21);
/// assert_eq!(result, 42);
/// ```
pub fn double(n: i32) -> i32 {
    n * 2
}

And here is that documentation example as ordinary, compilable code -- the assertion is the same one the doctest runs, so if double ever regressed, the docs would catch it:

pub fn double(n: i32) -> i32 {
    n * 2
}

fn main() {
    // this is precisely the body cargo test would execute from the doc example above:
    let result = double(21);
    assert_eq!(result, 42);
    println!("doctest body passed");
}

Inside a real doctest you have a few dials. A line beginning with # is hidden from the rendered documentation but still compiled and run -- perfect for boring setup (imports, constructing a value) that would clutter the example a reader sees. Marking the fence no_run compiles the example but does not execute it (handy for something that opens a socket or writes a file). Marking it ignore skips it entirely, and should_panic asserts it panics, mirroring the unit-test attribute. But the default -- compiled and run -- is the one you want almost always, precisely because it keeps your docs honest. Doctests are the reason Rust's ecosystem documentation is so reliably correct: the examples are not decoration, they are tests.

Running, filtering, and the parallelism trap

cargo test builds and runs the lot -- unit tests, integration tests, and doctests, in that order, with a summary for each. In daily work you rarely want all of them, so you filter by name substring: cargo test parse runs only tests whose name contains parse. You already saw -- --ignored for the slow ones. And because the harness captures everything a passing test prints, you use cargo test -- --nocapture when you actually want to see your println! output (the harness hides it by default so a green run stays quiet).

Now the trap I flagged earlier. Tests run in parallel by default, each on its own thread. That is wonderful for speed and merciless for sloppiness: if two tests both touch the same file on disk, the same environment variable, or any shared global state, they will race and fail intermittently -- the worst kind of failure, the one that only shows up on the build server at 3am. The fix is almost always to make each test self-contained: give it its own temporary directory, its own fresh data, no reliance on order. If a set of tests genuinely must run one at a time, you can force it with cargo test -- --test-threads=1, but treat that as a last resort and a bit of a code smell -- independent tests are better tests.

How Python and Go do it

A glance sideways sharpens the picture, since many of you came through the Learn Python Series. In Python with pytest, tests are functions named test_* in files named test_*.py, discovered by convention, and you assert with the plain assert keyword because pytest rewrites it to produce a helpful message:

# test_math.py -- pytest discovers files named test_*.py
def add(a, b):
    return a + b

def test_adds():
    assert add(2, 3) == 5

def test_raises():
    import pytest
    with pytest.raises(ZeroDivisionError):
        _ = 1 / 0

It is excellent, but notice the differences: pytest is a third-party install you choose, the tests live in a separate tree by convention rather than beside the code, and there is nothing quite like doctests-run-by-default baked into the standard workflow (Python has a doctest module, but few projects wire it into CI).

Go is philosophically the closest to Rust: testing is in the standard toolchain (go test), tests live in _test.go files next to the code, and you signal failure by calling methods on a *testing.T in stead of panicking:

// math_test.go -- go test discovers files ending in _test.go
package math

import "testing"

func Add(a, b int) int { return a + b }

func TestAdd(t *testing.T) {
    if got := Add(2, 3); got != 5 {
        t.Errorf("Add(2, 3) = %d; want 5", got)
    }
}

Go and Rust share the instinct that a language should ship its own test runner so the whole community writes tests the same way -- no framework wars. Rust pushes it one step further with doctests and with the unit/integration split enforced by where the file lives. Same spirit, a little more built in.

Wrapping up

So here is the whole picture. Rust bakes testing into the language and cargo, no framework required. Unit tests live in a #[cfg(test)] mod tests next to the code, can reach private items, and are stripped from your release build. You assert with assert!, assert_eq! and assert_ne!, prove deliberate panics with #[should_panic(expected = "...")], and use ? inside Result-returning tests. Integration tests live in tests/, compile as separate crates, and exercise only your public API -- guarding the contract while the unit tests guard the internals. And doctests turn the examples in your /// comments into a live test suite, so your documentation can never quietly lie. cargo test runs everything in parallel, which is fast but demands that your tests be independent.

Having said that, the deeper habit matters more than any macro: write the test while you write the code, not "later" (later never comes), and let a red test be the thing that tells you a change broke something, in stead of a user. A crate with a green cargo test is a crate you can refactor without fear -- and that fearlessness is most of why Rust is a joy to work in.

We have only tested with examples we dreamt up, though, one input at a time. That leaves an obvious question hanging: what about the inputs we did not think of? In the next episodes we start letting the machine invent inputs for us by the thousand, throwing pseudo-random and deliberately hostile data at our functions to find the edge cases our imagination missed -- and then we will start measuring how fast our code actually runs, rather than guessing. Testing proves correctness; next we go hunting for the bugs and the bottlenecks that ordinary tests never reach ;-)

Exercises

Write is_palindrome(&str) -> bool (ignoring case is optional) and a #[cfg(test)] module with assert! cases for a real palindrome, a non-palindrome, and the empty string.
Write a function that panics on a negative argument, then write a #[should_panic(expected = "...")] test proving it panics for the right reason -- plus a second, ordinary test proving it returns the correct value for a valid argument.
Write a test that returns Result<(), std::num::ParseIntError> and uses ? on a parse step, and add an #[ignore]d test with a documented reason next to it.

Thanks for reading -- writing tests is a habit, not a chore, so go make it one. De groeten! ;-)

@scipio