Learn AI Series (#135) - Building AI Teams and Processes

scipio(71)
Published in
StemSocial
Words
4771
Reading
22 min
Listen
Play
3M

Learn AI Series (#135) - Building AI Teams and Processes

variant-b-12-green.png

What will I learn

  • The distinct roles in an AI team and why they do not map cleanly onto the frontend/backend/QA boxes you already know;
  • the ML lifecycle as it really behaves -- a loop, not a line, that keeps dragging you back to the data;
  • model cards and datasheets: the boring documents that save you six months from now when a model starts misbehaving and nobody remembers why;
  • what to look for in an ML code review that ordinary software review sails straight past (data leakage being the big, silent killer);
  • the peculiar technical debt that ML systems rack up faster than normal code -- and why it breaks quietly instead of loudly;
  • and the organizational traps -- the proof-of-concept trap, the "just retrain" reflex -- that kill more AI projects than any bad algorithm ever did ;-)

Requirements

  • A working modern computer running macOS, Windows or Ubuntu;
  • An installed Python 3(.10+) distribution; the code here is plain Python plus a little numpy (pip install numpy) -- no GPU, no PyTorch, because today is about process, not tensors;
  • You've been through episode #34 (ML Engineering), #117 (ML System Design) and #119 (Experiment Tracking) -- and last episode's #134 on infrastructure economics, whose three exercises we settle right below.

Difficulty

  • Beginner

Curriculum (of the Learn AI Series):

Learn AI Series (#135) - Building AI Teams and Processes

I ended episode #134 on a deliberately unsatisfying note. We had spent the whole thing learning to calculate our way through infrastructure -- GPU rates, spot instances, the on-prem crossover, make-versus-buy reduced to arithmetic you can actually run. And then I pointed at the thing the spreadsheet cannot price: that every one of those clean decisions quietly assumed someone owned the shutdown script, someone chose the provider, someone was actually allowed to say "we build this in stead of renting it". Compute you can budget. The humans around the compute you cannot, and that is exactly where most AI projects go to die.

So today there is almost no math and no model at all. Today is about the messy, unglamorous stuff that decides whether all the technical skill this series has drilled into you actually ships anything: roles, lifecycle, documentation, review, debt, and the organizational landmines nobody warns you about. If you are a team of one -- and quite some of you reading this are -- do not tune out. Process is not bureaucracy for solo builders; it is insurance against your own forgetting. Here we go ;-)

Solutions to episode #134's exercises

Homework first, as always -- full code, no hand-waving.

Exercise 1 -- turn the estimator into a budget tool. The task was to take a fixed budget ceiling and, for every provider/instance in the rates table, print how many GPU-hours that budget buys (on-demand and spot), then declare which single option lets you train the longest.

def budget_hours(budget, use_spot=False):
    """For a fixed budget, how many GPU-hours does each option buy?"""
    rates = {                                      # per-GPU $/hr, on-demand, early 2026
        ("A100", "aws"): 4.00, ("A100", "gcp"): 3.67, ("A100", "lambda"): 1.25,
        ("H100", "aws"): 12.25, ("H100", "gcp"): 11.00, ("H100", "coreweave"): 3.00,
        ("L4", "aws"): 0.80, ("L4", "gcp"): 0.70,
    }
    best, best_hours = None, -1.0
    for (gpu, prov), rate in rates.items():
        r = rate * 0.3 if use_spot else rate       # spot ~ 70% off
        hours = budget / r
        if hours > best_hours:
            best, best_hours = (gpu, prov), hours
        print(f"{prov:>9} {gpu:<5} {'spot' if use_spot else 'on-demand':<9} "
              f"${r:>6.2f}/h -> {hours:>7.1f} GPU-h")
    print(f"LONGEST under ${budget:,.0f}: {best[1]} {best[0]}  ({best_hours:.1f} GPU-h)\n")
    return best

budget_hours(1000, use_spot=False)
budget_hours(1000, use_spot=True)

The one sentence: with a flat dollars-per-hour table the "longest" option and the "cheapest per hour" option are always the same thing -- lowest rate mechanically buys the most hours -- and they only diverge once you weight hours by throughput, because an H100 at triple the price but quadruple the speed finishes more actual work per dollar even though it buys you fewer raw hours on the clock.

Exercise 2 -- model your own crossover. Plug a realistic monthly query volume and today's API price into api_vs_custom, find the breakeven month, then redo it assuming traffic grows 15% month-over-month in stead of staying flat.

def api_vs_custom_growth(q0, api_cost, training_cost, infra_monthly,
                         months=24, growth=0.0):
    """Cumulative API spend vs. own-the-model, with optional monthly traffic growth."""
    api_cum, q, breakeven = 0.0, q0, None
    for m in range(1, months + 1):
        api_cum += q * api_cost                    # this month's API bill
        custom_cum = training_cost + infra_monthly * m
        if breakeven is None and custom_cum < api_cum:
            breakeven = m
        q *= (1 + growth)                          # next month's volume
    tag = f"{growth:.0%}/mo"
    print(f"[{tag:>7} growth] breaks even at "
          + (f"month {breakeven}" if breakeven else "never within horizon"))
    return breakeven

# 100k queries/month at $0.03, $5k to train, $500/mo to host
api_vs_custom_growth(100_000, 0.03, 5000, 500, months=24, growth=0.00)
api_vs_custom_growth(100_000, 0.03, 5000, 500, months=24, growth=0.15)

At flat traffic the custom model overtakes the API somewhere around month three; switch on 15% monthly growth and that crossover marches earlier, because every extra query you serve widens the gap the API is charging you per call while your training cost sits fixed. The lesson is not the exact month -- it is that growth is a thumb on the scale that always pushes toward owning the thing, and the faster you grow the sooner "rent" turns into "buy".

Exercise 3 -- make the idle guard real-ish. Rewrite the shutdown guard so it consumes a stream of simulated utilization readings across a whole week, kills the box after patience idle checks, and reports the dollars saved versus a machine left humming the entire week at $32/hr.

import numpy as np

def simulate_week(patience=6, low=0.05, check_every_min=10, hourly=32.0, seed=0):
    rng = np.random.default_rng(seed)
    checks = 7 * 24 * 60 // check_every_min        # one week of periodic checks
    idle_streak, minutes_billed, shut = 0, 0, False
    for _ in range(checks):
        if shut:
            continue                               # box is off -> stops billing
        # mostly idle, with an occasional genuine burst of work (~20% of checks)
        util = rng.uniform(0.4, 0.9) if rng.random() < 0.2 else rng.uniform(0.0, 0.03)
        minutes_billed += check_every_min
        idle_streak = idle_streak + 1 if util < low else 0
        if idle_streak >= patience:
            shut = True
    hours_billed = minutes_billed / 60
    full_week = hourly * 7 * 24
    saved = full_week - hours_billed * hourly
    print(f"patience={patience:>2}: billed {hours_billed:>5.1f}h "
          f"(${hours_billed*hourly:>6,.0f})  saved ${saved:>6,.0f}  of ${full_week:,.0f}")
    return saved

for p in (3, 6, 12):
    simulate_week(patience=p)

One sentence on choosing patience: set it so that patience * check_every_min comfortably exceeds the longest legitimate idle gap in your workload -- the pause between epochs while validation and checkpointing run, say -- otherwise the guard will cheerfully execute a job that was merely catching its breath between rounds. Right, homework done. Now let us talk about the people who were supposed to write that shutdown script.

The roles: who actually does what

In traditional software the roles are worn smooth by decades of use -- frontend, backend, DevOps, QA, everyone knows roughly where the lines are. AI teams are younger, the titles are mushier, and the responsibilities overlap in ways that cause real friction. Here is the honest breakdown.

Data Scientist. Explores data, builds prototypes, runs experiments. Lives in notebooks, fluent in statistics and visualization, brilliant at finding signal and turning a business question into something measurable. The classic weakness: produces code that sings in a notebook and falls over the instant it meets production traffic.

ML Engineer. Takes that prototype and makes it survive -- training pipelines, serving infrastructure, performance work. This is the bridge between "it worked on my laptop" and "it serves a million users". Everything we slogged through in episodes #117-#126 is this person's daily life.

Data Engineer. Builds and guards the pipelines that feed the models -- ETL, data quality, feature stores, versioning (episode #118 was the technical half of this job). Underinvest here and your ML engineer is quietly training on stale or corrupted data and nobody notices until the metrics fall off a cliff.

MLOps Engineer. Owns the operational surface: CI/CD for models (episode #124), monitoring (episode #123), rollouts, drift alerts. The role exists because shipping a model is genuinely not the same problem as shipping a web service -- models rot, and someone has to watch them rot.

Research Scientist. Reads papers, implements new architectures, pushes the frontier. Mostly a luxury of larger labs; in a small team the data scientist or ML engineer just wears this hat on Fridays.

The single most common staffing blunder -- and I have watched it play out more than once -- is hiring a pile of data scientists and no engineers. You end up with a team that can prototype anything and ship nothing, a beautiful gallery of demos that never reach a user. As a rough starting ratio for a product team, think something like one data scientist to two ML engineers to one data engineer. Having said that, the ratio matters far less than the principle: prototyping capacity without shipping capacity is a very expensive way to produce slide decks.

The ML lifecycle: it is a loop, not a line

Software has a development lifecycle. ML has one too, but the shape is different -- it curls back on itself, because models degrade and data moves. Six stages:

  1. Problem framing. Translate the business problem into an ML problem. "Reduce churn" becomes "predict churn probability per customer over the next 30 days." Most failures are conceived right here -- a mis-framed problem produces a technically flawless model that solves nothing anyone asked for.
  2. Data collection and understanding. What exists, what is missing, what is the quality, do you even have labels or must you create them? This phase always takes longer than the plan says.
  3. Experimentation. Prototypes, approaches, held-out evaluation. The "fun" part the tutorials obsess over -- and maybe 20% of the real work.
  4. Productionization. Turn the winning notebook into a reliable pipeline, build serving, handle the edge cases. Reliably 2-5x longer than the experimentation it follows.
  5. Deployment and monitoring. Ship it, then watch it. Does it behave in production like it did in evaluation? Is there drift? Failure modes you never imagined?
  6. Iteration. It degrades, requirements shift, new data lands. Back to stage 2 or 3. This is not failure -- it is the job. ML systems are living systems, and pretending otherwise is how you get blindsided.

A tiny state tracker -- the kind of unglamorous scaffolding teams actually build so nobody has to ask "wait, which stage is the churn model in?":

from dataclasses import dataclass, field
from datetime import datetime
from typing import Optional

@dataclass
class ModelLifecycle:
    """Track a model through its lifecycle stages."""
    model_name: str
    problem_statement: str
    owner: str
    created: datetime = field(default_factory=datetime.now)
    stage: str = "framing"   # framing, data, experiment, production, deployed, retired
    metrics: dict = field(default_factory=dict)
    data_version: Optional[str] = None
    model_version: Optional[str] = None
    notes: list = field(default_factory=list)

    def advance(self, new_stage, note=""):
        self.notes.append(f"[{datetime.now():%Y-%m-%d}] {self.stage} -> {new_stage}: {note}")
        self.stage = new_stage

    def record_metric(self, name, value, split="test"):
        self.metrics.setdefault(f"{split}/{name}", []).append(
            (datetime.now().isoformat(), value))

    def summary(self):
        print(f"Model: {self.model_name} (stage: {self.stage})  owner: {self.owner}")
        print(f"Problem: {self.problem_statement}")
        for key, values in self.metrics.items():
            ts, val = values[-1]
            print(f"  {key}: {val:.4f}  ({ts[:10]})")

churn = ModelLifecycle("churn-predictor-v2",
                       "Predict 30-day churn probability for active subscribers",
                       owner="ml-team")
churn.advance("data", "Historical data extracted, labels verified")
churn.advance("experiment", "Baseline gradient-boosted tree running")
churn.record_metric("auc_roc", 0.847)
churn.record_metric("precision_at_10pct", 0.72)
churn.summary()

It looks almost too simple to bother with. That is the point -- the teams that skip this are the ones where, three months in, nobody can say with a straight face which model is live, what it was trained on, or when it last got looked at.

Model cards: documenting what you built

A model card is a short, standardized document describing a model: what it does, how it was trained, where it works, and -- the part everyone skips and everyone regrets skipping -- where it fails. Google put the idea forward back in 2019 and it has since become basic hygiene for responsible deployment.

Why bother? Because six months from now, when the model starts misbehaving at 2am, someone needs to know what data it saw, what its known limits are, and what evaluation was actually done. That someone might be a colleague who has never seen the code. More likely it is you, having comprehensively forgotten every detail. Write it down while you still remember.

model_card = {
    "model_name": "churn-predictor-v2",
    "version": "2.1.0",
    "date": "2026-02-15",
    "type": "Binary classification (gradient-boosted trees)",
    "description": "Predicts probability of customer churn within 30 days.",

    "training_data": {
        "source": "internal CRM database, Jan 2023 - Dec 2025",
        "size": "245,000 customers, 18 features",
        "label_definition": "Customer inactive for 30+ days after observation date",
        "known_gaps": "Missing partner-channel signups before 2024",
    },
    "evaluation": {
        "test_set": "Jan-Mar 2026, 32,000 customers (temporal split)",
        "auc_roc": 0.847,
        "precision_at_10pct_recall": 0.72,
        "f1_score": 0.68,
    },
    "intended_use": "Prioritize retention outreach for at-risk customers",
    "out_of_scope": "Do NOT use for credit decisions or service denial",
    "limitations": [
        "Lower accuracy for customers with < 3 months history",
        "Not validated on enterprise accounts (trained on consumer only)",
        "Degrades if the pricing structure changes significantly",
    ],
    "ethical_considerations": [
        "Age and location features may introduce demographic bias",
        "Regular fairness audits required across customer segments",
    ],
    "maintenance": {
        "retraining_schedule": "Monthly on a fresh 3-month window",
        "monitoring": "Daily AUC tracking, weekly drift detection",
    },
}

Every field earns its place, but the limitations and out_of_scope sections are the ones that actually save you -- they tell a future user what not to trust the thing for. A model card without a limitations section is a sales brochure, not documentation. And because a document nobody validates rots into fiction, it is worth a five-line guard that simply refuses to let an undocumented card through review:

REQUIRED = ["model_name", "version", "training_data",
            "evaluation", "intended_use", "limitations"]

def validate_card(card):
    missing = [k for k in REQUIRED if not card.get(k)]
    if missing:
        raise ValueError(f"model card incomplete -- missing: {missing}")
    if not card["limitations"]:
        raise ValueError("a model card with an empty 'limitations' list is a lie")
    print(f"model card OK: {card['model_name']} v{card['version']}")

validate_card(model_card)

Wire that into CI and "we will document it later" quietly stops being an option -- the build fails until the card is honest. Nota bene: the same idea applies to datasets, where the equivalent artifact is called a datasheet (origin, collection method, consent, known biases). Same discipline, different noun.

Code review for ML: beyond syntax

Standard code review catches bugs, style slips, broken logic. ML review needs all of that plus a handful of checks that ordinary review sails straight past -- and the nastiest ones are the ones where the code is perfectly correct and the reasoning is wrong.

Data leakage. Is test data bleeding into training? Are features computed using information from the future? Was the scaler fitted on the whole dataset in stead of just the training fold? This is the most common and most devastating ML bug in existence, and it slides past normal review precisely because the code looks fine -- there is no crash, no exception, just a validation score that is a beautiful, seductive lie.

Reproducibility. Are the random seeds set? Run it twice, do you get the same answer? Can a colleague rebuild your result from the code and config alone? Episode #119 handed you the tools; review is where you enforce that they were actually used.

Metric selection. Is the metric right for the business problem? Accuracy on an imbalanced dataset (episode #13, forever haunting us) passes review and ships a useless model. The reviewer has to understand the problem, not just read the diff.

Training-serving skew. Does the feature computation at training time exactly match the one at serving time? "days_since_last_purchase" computed one way in the batch pipeline and another way at inference -- a timezone here, a null handled differently there -- and your live model silently sees features it was never trained on.

review_checklist = {
    "data_leakage": [
        "Preprocessing fitted ONLY on the training split",
        "No future-derived features in a time-series context",
        "Target variable not accidentally leaking into the features",
        "Holdout set truly held out (used for no decisions at all)",
    ],
    "reproducibility": [
        "Random seeds set for every source of randomness",
        "Data version pinned (a hash or a snapshot date)",
        "Dependencies pinned (requirements.txt with versions)",
        "Hyperparameters logged, not buried in a comment",
    ],
    "evaluation": [
        "Metric matches the business objective",
        "Realistic split (temporal for time-series, not random)",
        "A baseline to compare against",
    ],
    "production_readiness": [
        "Training and serving feature logic are identical",
        "Edge cases handled (nulls, out-of-range, unseen categories)",
        "Model size and latency inside the deployment budget",
        "Monitoring hooks actually wired in",
    ],
}

def review_gaps(pr_claims):
    """pr_claims: the set of checklist items the author swears they handled."""
    for area, items in review_checklist.items():
        missing = [it for it in items if it not in pr_claims]
        status = "OK" if not missing else f"{len(missing)} UNCONFIRMED"
        print(f"{area:<20} {status}")
        for it in missing:
            print(f"    - {it}")

review_gaps(pr_claims=set())   # nothing confirmed yet -> the full checklist lights up

Run that against an empty set and every item lights up red, which is exactly the state every pull request starts in. The checklist is not there to slow you down -- it is there so the leakage bug gets caught in review, where it costs a comment, in stead of in production, where it costs a quarter.

Technical debt that breaks quietly

Google's much-cited 2015 paper, "Hidden Technical Debt in Machine Learning Systems", named something every practitioner feels but few articulate: ML systems accumulate debt faster than ordinary software, and the debt is far harder to see. The reason is that most of it breaks silently.

Data dependency debt. Your model leans on an upstream pipeline. That pipeline changes its schema one Tuesday. Your model keeps running, keeps returning numbers, and quietly serves garbage until accuracy tanks days later. Code dependencies break loudly -- an import fails, a build goes red. Data dependencies break in a whisper.

Configuration debt. The sheer number of hyperparameters, feature flags, data paths and environment knobs in an ML system is enormous, and a single wrong one -- a stale data path, a learning rate off by a zero -- produces a model that trains successfully and performs terribly. It does not crash. It just gets worse, and you go looking for a bug in the model when the bug is in a YAML file.

Experimental debt. Every untracked experiment, every "temporary" hack that put down roots, every model variant living in someone's home directory but not in the registry. Over a year the team forgets what was tried, what worked, and why any given decision was made -- institutional amnesia with a loss curve.

Feedback-loop debt. When a model's own predictions shape the data it is later trained on. A recommender that only surfaces popular items gathers data proving popular items get clicked, which trains the next model to push popular items even harder -- a system optimizing itself into a filter bubble, one retrain at a time.

The cure is not zero debt -- that is a fantasy. It is making debt visible and paying it down on purpose: track experiments (episode #119), version your data, document decisions, review on a cadence. A quick sketch of the "quiet" part made loud -- a guard that screams when an upstream schema drifts before the model eats the bad data:

def check_schema(expected, incoming):
    """Fail loudly the moment an upstream data contract changes."""
    missing = set(expected) - set(incoming)
    added = set(incoming) - set(expected)
    if missing or added:
        raise RuntimeError(f"schema drift! missing={missing or '-'} added={added or '-'}")
    print("upstream schema OK")

expected_features = {"age", "tenure_months", "monthly_spend", "n_logins_30d"}
todays_feed = {"age", "tenure_months", "monthly_spend"}   # someone dropped a column
check_schema(expected_features, todays_feed)              # -> RuntimeError, not silent rot

Eight lines turns a silent, week-long accuracy bleed into a loud crash at 9am on the day it happens. That trade -- loud-and-early over quiet-and-late -- is the whole philosophy of managing ML debt in one idea.

The organizational challenges (the truly hard part)

The technical failures are the ones you can fix. The people failures are the ones that sink projects.

Expectation management. Stakeholders expect AI to be a magic wand -- "just train a model to predict X". They do not see the data, the labels, the evaluation, the iteration, the endless maintenance. Setting honest expectations upfront ("three months to a prototype, six to production, then it never really stops") is unglamorous and it is the single highest-leverage thing you can do to keep organizational trust intact.

The proof-of-concept trap. A data scientist builds a dazzling demo in two weeks. Management sees it, greenlights production, and budgets none of the six months of engineering needed to make it reliable. The demo becomes production, held together with duct tape and optimism, and the team spends the next year firefighting in stead of improving. I have seen this exact movie more than once, and it never ends well.

Incentive misalignment. Data scientists are rewarded for trying new things; ML engineers are rewarded for keeping things stable. Those goals pull against each other by design. A healthy team names the tension out loud and builds process that balances novelty against reliability, in stead of pretending everyone wants the same thing.

The "just retrain" reflex. A model degrades and the instinct is to retrain on fresher data. But if the real cause is a data-quality issue, a concept shift, or a feedback loop, retraining on bad data makes it worse. Diagnosing why a model degraded is harder than training a new one -- and that diagnostic skill is precisely what separates a senior practitioner from a junior with a .fit() habit.

The solo developer version

Working alone -- as quite some of you are -- the same principles apply, just scaled down:

  • Document your decisions even though nobody else will read them (future you is somebody else, and a forgetful one);
  • use experiment tracking, even plain local MLflow, in stead of trusting your memory;
  • write a model card for anything you put into production, personal projects included;
  • version your data, not just your code;
  • set up the crudest possible monitoring -- a cron job that emails you when a metric drops is infinitely better than finding out from a user.

The solo builder's signature failure is the "it is all in my head" trap. Everything runs fine until you take a two-week break and come back to a system you no longer understand. Process is not corporate ritual. It is a note passed forward to a version of you who has forgotten everything -- and that person always shows up eventually.

Round-up

  • AI teams need distinct roles -- data scientist, ML engineer, data engineer, MLOps -- and hiring only scientists builds a team that prototypes brilliantly and ships nothing;
  • the ML lifecycle is a loop (framing -> data -> experiment -> production -> deploy -> iterate), and productionization reliably costs 2-5x more than the experimentation it follows;
  • model cards (and dataset datasheets) document what a model does, how it was trained, and above all where it fails -- validate them in CI so they cannot rot;
  • ML code review adds data leakage, reproducibility, metric fit and training-serving skew on top of ordinary review, and leakage is the silent killer;
  • ML technical debt accrues faster and breaks quietly -- data dependencies, configuration sprawl, untracked experiments, feedback loops -- so the whole game is making it loud and visible;
  • the organizational traps (magic-wand expectations, the POC trap, incentive clash, the "just retrain" reflex) kill more projects than any algorithm;
  • and solo builders need every one of these disciplines at smaller scale, because future-you is the least reliable teammate you will ever have ;-)

Exercises

Three to gnaw on before next time -- and, as always, I will walk through full solutions at the top of the following episode.

  1. Grow the lifecycle tracker a spine. Extend ModelLifecycle so that advance() refuses illegal transitions -- define the legal order (framing -> data -> experiment -> production -> deployed -> retired), allow going backwards to "data" or "experiment" (that is iteration, and it is legal), but reject nonsense like jumping straight from "framing" to "deployed". Add a days_in_stage() helper and a one-line health check that flags any model stuck in "experiment" for more than, say, 60 days. One sentence on why "stuck in experiment" is a smell worth alerting on.

  2. Write your own model card -- and break it. Take any model you have actually trained in this series (the churn tree, a CNN from #46, anything) and fill in a real model_card dict for it, honestly. Then run validate_card against it. Now deliberately empty the limitations list and confirm the validator rejects it. Write two sentences: one naming a genuine limitation of your model that you would not have written down without the card forcing you to, and one on why that omission would have bitten a future user.

  3. Simulate feedback-loop debt. Build a tiny recommender that always recommends the current top-3 most-clicked items, then simulate 20 rounds where each round's "clicks" are drawn mostly from whatever was recommended (with a small chance of a random discovery). Track how the diversity of recommended items changes over the rounds and show the system collapsing toward a handful of winners. One sentence on the cheapest intervention (hint: think exploration, and cast your mind back to episode #103) that would keep the catalogue from strangling itself.

We have spent this whole series building the pieces -- the math, the models, the pipelines, the infrastructure, and today the humans and process that hold it all together. So here is the question I want you carrying into next time, and it is a big one: could you take everything from the last hundred-odd episodes and stand up a single, coherent, end-to-end system -- data in one side, a monitored, documented, retrainable model serving predictions out the other, with the roles and the review and the lifecycle all wired in? Not a notebook. A platform. That is where we head next, and we are going to build it ;-)

Bedankt en tot de volgende keer -- go write the model card you have been putting off!

scipio@scipio

stem
stemsocial
steemstem
python
programming
via hiveblog
This post was published via hiveblog. Ecency is not the originator or editor of this content and displays it for discovery purposes only.
24
Learn AI Series (#135) - Building AI Teams and Processes | Ecency