Inside the Sausage Factory

Yesterday, I saw a question over on Ask MetaFilter entited "How is the software sausage made?" and asking about different folks' software development methodologies and processes. I decided to share the way that things were done at Canonical, because that's where I learnt a lot of development best-practices (which I continue to keep practising even now). It turned out to be quite long but interesting, but I thought I'd share it here.

I'm going to tell you about the previous place I worked at, because at my current (contracting) job things are a bit more start-up-ish and free flowing. Genuinely, Canonical — specifically teams that grew out of the Launchpad team as-was — has some of the best development methodology going, and I constantly refer back to it. The last project I worked on before I left was this one; luckily that team was entirely ex-Launchpadders, so we kept the same development flow.

So, the tools first:

  • Bug tracking / feature requests / code reviews / specifications ("blueprints"): Launchpad. Old and not as widely used as Github, but my god was the bug tracker a good one (disclosure: I wrote a lot of it, so I'm tooting my own horn here).
  • Work tracking: Kanban. Brilliant for coordinating and visualising work, if used correctly. The backlog would often fill up because we a) couldn't see it by default and b) had too much other work on to deal with it all, but mostly this was a great resource for us.
  • Source control: Bazaar. Another Canonical-built too. I've a soft-spot for it because I got so used to using it. Just as good as Git or Hg in my opinion, but sadly lost the battle to be a front-line VCS.
  • Continuous integration: Jenkins (or, in days of yore, Buildbot) and Tarmac.

The workflow, which was as close to Lean / Kaizen as we could get, went something like this:

  1. Grand Vision is expounded by Self-Appointed Benevolent Dictator for Life (sabdfl).
  2. Team iterates over Grand Vision, working out the details, breaking it up into user stories, and the user stories into development / design tasks
  3. Team agrees on targets for the feature. The key parts of this are the user stories and a "We'll know we're done when" for each of those and for the feature as a whole.
  4. [Optional] Estimations done for each task. This didn't always happen, but when it did we were much better at giving realistic expectations to the sabdfl. Since Ubuntu has a 6-month release cadence this was often about what parts of the feature we could do in those six months, not how long the feature would take us. Lean methodology came in here a lot of the time: focusing on an MVP for a feature and then building from there so that we didn't release half-broken things.
  5. Cards for each task go on the Kanban board. At one point we tracked everything with Launchpad and wrote a tool that synced Launchpad Bugs to the Kanban board, but that became a headache. Kanban was simpler for feature development tracking.
  6. Development: A developer takes or is assigned a task card — which should be Ready To Code and ideally about one day's worth of work for one dev — and works on it until done. Methodology (for ex-LP'ers) is strictly TDD. You won't be able to land code unless there are tests, as a rule.
  7. Code review (using Launchpad). One review is necessary per branch that's ready to land. The branch can't land until the reviewer approves it and the tests pass. For LP, we had a fairly detailed review checklist; for other projects the review checklist wasn't necessarily codified, but was similar.
  8. Landing (automated, using Tarmac). If the tests don't pass, the branch won't be merged into the main line of development, and the developer is responsible for fixing the tests.
  9. Continuous Integration: Jenkins runs a set of acceptance and integration tests to ensure everything works end-to-end on the updated main line branch. If there's a failure here, the developer is responsible for fixing it. Kaizen was part of our methodology here: big failures were red-button issues, meaning that the whole team dropped their current work to focus on fixing the failure.