Release Management and COVID-19

I stumbled upon a Dave Farley video, as part of his Continuous Delivery series, where he talks about Release Management. While talking about how science impacts the decisions that one makes to release software, Farley quotes Richard Feynman:

“It doesn’t matter how intelligent you are, if you guess and that guess cannot be backed up by experimental evidence – then it is still a guess!”

Now, let’s think about where we are today. COVID-19 is spreading fast throughout the world, with several million confirmed cases and several thousand fatalities. If you “guess” and release a vaccine that doesn’t work, it’s still a guess. What do you need to prove that the vaccine will work? Experimental evidence. For vaccines, that comes in the form of clinical trials. But, this effort will require “release management” on a global scale, and the cost of failure will be lives. Back when I worked at General Electric, in their Medical Systems division in Waukesha, WI – the motto of the Medical Systems division was “we are in business to save lives”. The flip side of that – “we don’t do anything to risks lives” – can be aligned to release management. “We don’t do anything to <fill in the blank>” can be your motto while releasing software to hundreds of thousands of customers.

So, let’s apply Farley’s process to release management of the COVID-19 vaccine –

• Hypothesis – People will be able to live without fear of contracting the coronavirus. (I chose that over “our software will continue to work” because it’s congruent conceptually)
• Measurements – We need to consider the following for the clinical trials:
  • Automated Testing – how are we going to test this vaccine?
  • Automated Deployment – how are we going to deploy the vaccine? is it a shot or medicine taken orally?
  • Configuration Management – what control groups (environments) are we releasing this vaccine into?
  • Monitoring & Health-Checks – how are we monitoring the results of the vaccine during clinical trials? how are we collecting data for future analysis? any “business” impacts of the vaccine?
• Feedback – once this vaccine is released for clinical trials:
  • Continuous Integration & Continuous Delivery – how do we evaluate the release of the vaccine and improve delivery?
  • Dashboards – how do we get clear indications of the state of the vaccine?
  • Canary releases – can we release to a smaller, safer subset (control group) to determine the impact, if any, on people? (users)
• Controlling Variables – some of the key variables we can control:
  • Make Changes in small steps – unfortunately, if your vaccine doesn’t work, you will have to go back to the drawing board, but you can restart the cycle if you make small changes
  • Good Configuration Management
  • Reliable Delivery (CD)
  • Quality
  • Monitoring

Netflix has a novel way (pun not intended) of canary releases – they release software in a geographical area when it’s 2 AM in that time zone. This allows Netflix to determine a “canary index” of releases as the release stays in production during daytime in that timezone. This allows the company to determine whether to pull that release out or distribute it to the other timezones at the same time, 2 AM. This is Netflix’s version of testing in production.

Unfortunately, with the number of lives lost during this pandemic, public sentiment around the globe will push for “testing the vaccine in production” more than ever. If you follow the steps above, you should be able to control the variables enough in your environment to allow the vaccine to be distributed widely, safely, and effectively in a relatively short amount of time.

LinkedIn – linkedin.com/in/sagarkarma

Twitter – @sagarvnvdevops