Getting to Continuous Delivery
Continuous Delivery allows us as developers to respond quickly to feedback to ensure we are meeting the business needs. Being able to respond quickly to changes the business needs without putting underdo strain on our resources will help us deliver value to the company and by extension our customers.
Definitions
Continuous Delivery
Continuous delivery is when the code in the main or master branch is in state where it can be deployed to production at any point. When it is deployed is up to the business. It could be once a week, once a month, or ten times a day. The software is manually verified at some point during this process and approved for production.
Continuous Deployment
Continuous deployment is any change checked into main or master branch is pushed to production as soon as it passes through the gates automatically. The software is verified through automated tests during this process and is approved for production.
Delivering Software Changes Quickly
Quickly can mean something different depending on the person being asked. For the purpose of this post quickly means less than two weeks.
Development Team
The development team is the group of individuals responsible for making and supporting the necessary changes to the software. This team consists of multiple developers, database developers, business owners, quality assurance engineers and business analysts.
Support Staff
Support staff is responsible for ensuring the pre-prod and production environments are up and running. They approve and deploy all changes to those environments. Support staff consists of web administrators, database architects, integration specialists, and network engineers.
Feedback
For the purposes of this post, feedback can mean any of the following.
Feature Request
A new feature or a new application is created to meet a specific business need.
Feature Improvement
An existing feature is enhanced to better support the business or eases the burden of our users.
Bug Fix
Functionality does not work as the business defined it or performing a certain action causes an error preventing the user from continuing with the application.
Code Analysis
Ensure code meets the styling and naming standards established for the entire company.
Automated Tests
Unit tests, integration tests, UI automation and performance tests fall under this umbrella. Automation tests give the development team confidence a change introduced to the code does not break other functionality.
Continuous Delivery vs Continuous Deployment
Continuous deployment is the end goal of continuous delivery, but requires a software shop to be near perfect at continuous delivery. At the time of this writing a lot of is still need to get through the obstacles to get to continuous delivery. Until those obstacles are overcome continuous deployment will remain the idealized goal. The purpose of this post is to get to continuous delivery.
Current Practices
Below is a brief description of the non-continous delivery development practices I have experienced working for various companies. This section is included with the goal of establishing a baseline to improve from.
Development work on the source code is currently done in a development branch. Once the change has been completed and vetted by the developers the change is merged into a master or main branch. All changes checked into the development branch are automatically deployed to development web and application servers. All changed merged into master are automatically deployed to test web and application servers. Changes are queued in the main or master branch until the business decides to push to production. This can result in a “big bang” deployment.
Development work on the database is currently done in the development database. Once code is ready to be pushed up to system test the developer will manually push the changes to the system test database.
Pushing to the higher environments is very manual. A database developer will run a schema comparison tool between system test and integrated or integrated and production. The schema comparison tool will generate a script to get the higher environment inline with system test. That script is submitted to the database architects to be run. Meanwhile MSDeploy or MSI packages are copied into a “Gold” folder where support staff will promote the code to the higher environments.
Some sort of paperwork must be submitted in order to push to the higher environments. This could be as simple as an email or a small form on a web page for pre-prod to a complex series of approvals for production.
The manual process is a result in conflicting goals. The Development team’s goal is to get changes to production as quickly and efficiency as possible. Support staff must ensure development team’s changes do not “break” production or cause unforeseen issues. Each development team can be responsible for one or more applications. As a result, deployments are manual. Database architects review deployment SQL scripts and deploy them. Web administrators run a script to push web applications to production.
Current Practice Pain Points
The current practice works. It has worked for a while, and compared to a lot of other places I have been is really top notch. But it does have a numerous pain points. Overcoming these pain points is not a simple matter of going faster but a complete mind shift over to continuous delivery.
Manual Promotion
Most of the work the support staff does to handle promotion to integrated and production can be summed up with the phrase “F5 Monkey.” All they do is hit a single button to promote the changes. This is a gross oversimplification, the point being a lot of manual work, which can be automated.
Support Staff Resources
The manual process puts a strain on support staff resources. Each development team is responsible for one or more applications. If each development team deployed a single application once a week someone from the support staff would have to work late multiple times a week. As a result of development practices mentioned in the previous section, “big bang” deployments have to occur during off hours, resulting in a strained work-life balance for the support staff.
Inconsistent Database Development Practices
Each development team does their database development differently. Some teams place their database into source control using a dbproj file. Other teams do not bother at all with source control. Some teams require the database developer do all the database changes while others let any developer make the changes.
Shared Development Database
Developers make source code changes locally using their own instance of IIS. However all database changes are made on a shared database. A simple change such as renaming a column could result in every developer being forced to stop until the code is fixed.
Manual Promotion of Database Changes
Manually generating a script for each environment is open to numerous problems. Each promotion is the first time ever the script has been run. It has not been tested in the lower environments. An unknown number of errors can occur or changes can be flat out missed or even forgotten.
Code Promotion and Branching Strategies
Having an everlasting development branch and then merging into master branch opens the door for many issues. Developer A commits a change to the development branch. They discover a bug in their code prior to going to master. Meanwhile Developer B commits a change to development branch. Developer A fixes their bug and is ready to push to master but sees Developer B’s changes. Developer A must ensure Developer B’s changes can either go to master or are in a different section of code that doesn’t affect his changes and can be promoted later. This scenario results in the following:
- Code being promoted before it is ready
- Unintended blockers. Have to wait until both developer’s code is ready to be promoted
- Cherry-picking changes to promote. The code deployed to development is not the same code promoted to test and higher.
- Slows down refactoring and other behind the scenes improvements
- The dreaded statement being uttered during testing, “it works on my machine”
Changes to get to Continuous Delivery
Getting to continuous delivery is not as simple as announcing “we are now a continuous delivery shop.” It requires multiple changes to the existing process. Each of these changes should address one or more of the listed pain points.
Master or Main to Production
Adopt the mindset that the Master or Main branch should always be ready to deploy to production. There are number of development practices which can be adopted to ensure this is the case. Those development practices can take up an entire document just by itself and will not be addressed in this document. At the very least, having this mindset will spur a lot of discussion during development.
Make use of Feature Branches
All development work should be occur on feature branches instead of the main or master branch. Once the feature branch is ready to be promoted to production the changes will be merged into main or master.
Consistent deployments across all environments
What is deployed to the development servers should be the exact same thing deployed to test, pre-prod and production. Be it an MSDeploy package, an MSI or a database change. This will ensure changes have already been tested three times before going to production.
Move to a dedicated database for development and use source control
Developers make source code changes locally and test using a local instance of IIS. It makes little to no sense to have a shared database model. It is the same as having the source code on a file share and changing it up there. Doing this will solve several problems.
- Help support feature branches. A required database change for a specific feature will only go in the feature branch and be shared between the developers working on the feature.
- Encourage database experimentation. Changing the database locally will not cause developers to stop working until code is changed to reflect database
- Keep track of who made a database change and when.
- Help automate the promotion of changes to higher environments. Numerous tools exist that can take a source controlled instance of a database and promote the differences to a higher environment.
- Allow the source code database to be analyzed via static analysis tools. This will help the database architects gain trust in the overall process.
Adopt a consistent database change process
The version of the database in source control will become the truth center. Create a package representing a snapshot of the database under source control and promote that through the various environments.
Automation
Automate where it makes sense. Instead of having support staff manually pushing changes use some sort of tool to allow them to do that.
Obstacles to Continuous Delivery
Changing the existing process will result in multiple obstacles. Here are some of the foreseen obstacles.
Push Back
This is going to result in “a lot of moved cheese” for development teams, support staff, and the business.
Tooling Cost
In order to get to continuous deliver we must either purchase or internally develop a lot of tools. An example would be Red Gate’s SQL Source Control.
Resource Cost
It will take time for each developer to set up his or her own instance of SQL Server. It will take time to setup automated deployments for each application.
Training Cost
Any new technique or tool will require some degree of training. It could be as simple as an online video or course to getting a group of people in a room and getting everyone on the same page.
The Fear of the Black Box
Manual changes and manual deployments result in more control. That control is going to have to be given up to a degree to automated processes.
Tooling
This post intentionally left tooling vague. It is the feeling of this author it is much more important to focus on the process and the overall goal instead. With that said, when it comes down to it, which tool or tools chosen is not all that important. With the caveat the tool or tools chosen help meet all the goals and updated processes detailed in this document quickly without a lot of customization.
Adoption Phases Sample
- Make database process improvements
- All developers get their own instance of SQL Server
- Database is placed under source control
- Database is analyzed by static analysis tools
- Promotion is from the source controlled version of the database
- Deployment Process Improvements
- Adopt the use of feature branches
- Adopt non-braking development practices
- The same code / database packages deployed to development are the same ones deployed to test, pre-prod, and production
- Automate Deployments to Pre-Prod
- Code and database deployments to integrated become automated through a tool.
- The tool will allow support staff to approve the changes
- Automate Deployments to Production
- The same process / tool used to get to pre-prod will be used to get to production
- The tool will allow the business to sign off on the changes