Netflix build process

Netflix are well known for their excellent deployment capabilities.

In this article they go over their build process and tools, from compiling code all the way to production, and presenting their current challenges and future plans.

A highly recommend read for anyone who is interested in doing DevOps the right way.

Advertisements

Branch in Code (a.k.a. Config Flags)

Inspired by an article on CM Crossroads I stumbled upon lately, I’d like to give a high-level overview on this concept, commonly use in web site developments and can also be applied to cloud applications. But most people (like me) who work with ‘installable’ software never heard of it, and if they did, they regard it as an alien concept that just can’t work.

The short version is this:

  • Your software is a service. There is only a single deployment of that software, which means a single relevant codebase.
  • Hence there is no point of doing branching and merging in your source control. Everyone commits to the mainline
  • Instead of branching, developers wrap every piece of new code with some condition that can be controlled externally (e.g. a flag in a config file, user properties, server environment, etc.)

This way you can “switch versions in runtime”. Which lets you do things like:

  • Get rid of feature branches and merges – just deploy the “dark code” (partial/untested code) for a new feature ahead of time and, when the feature is ready, you can easily turn it on (and off if problems appear)
  • Roll out an update gradually and selectively
  • Rollback an update in a matter of seconds
  • Offer different UI experience to different users
  • Test new code on production servers (by conditioning based on e.g. user id or IP range)
  • And more…

Challenges:

  • The code can become somewhat “ugly”. Although many of those flags eventually get removed from the code once they don’t have a use in production.
  • ‘config file explosion’ – there are several different approaches regarding controlling the amount and lifespan of configuration flags
  • ‘mysterious flags’ – here too, there are some ideas about maintaining proper documentation as well as performing cleanup/refactoring as needed.

 

Feature-Based Development in a Version-Based Environment

I work for a company which uses Version-Oriented development. This means that we decide in advance which new features and enhancements to include in each planned version release. Hence, at any given time, most developers are working toward one specific version (with the occasional hot-fixes or patches for past versions).

However, we have some customers who were promised a certain set of  features, which we would like to deliver as soon as possible without having to wait for a formal version release. How do we develop them, and how do we release them?

Surprisingly, one of the toughest problem is “how to name such versions?”. Do our SCM tools support non-standard names? what version ID do we want to show the customer? will we be able to upgrade it to an “official” version in the future?

We recently implemented a simple, but effective solution for this issue. Up until now we used 2 numbers to represent our official versions (e.g. 5.0, 5.1). Now, we started using 4 numbers (e.g. 6.1.2.0). The idea is as follows:

  • The first two numbers are for major releases, just as it was before.
  • The third number represents a minor version – containing relatively small amount of new features or enhancements, and has a shorter release cycle.
  • The fourth number is for feature-specific versions.

Admittedly, the fourth number may cause some confusion, since for example, 6.1.2.4 doesn’t necessarily contain the features from 6.1.2.3. But since each of them it targeted for a different customer, it shouldn’t be a problem. And we do know that both features will be included in the next minor release, 6.1.3.0.

One important thing is to record the customer to which a feature-specific version ID is assigned, in a visible way. Currently we do it via a shared document, but we will implement a better system in the future (such as using a Version Object in the change tracking tool, instead of simple text field).

 

So far the system works well. But as usual, there is always room for improvement…