Build a VM Test Lab using Unraid - Working From Home Part I
Understanding how hypervisors work is key to setting up a test lab on a home server. In this article, read about lessons I learned setting up a VM test lab on Unraid.
As stated in the previous article, DevOps is the answer to the question, "what do I have to change in my process to be able to and want to deploy to production 10 times a day with zero downtime?" The number 10 was not just some random number picked from a hat. We had a random number generator do that for us. In all seriousness, that was chosen because that means in a typical eight-hour day, that means going to production every 45 or so minutes.
Framing that definition in minutes really throws it into a new perspective. This means the deployment process has to scale really well, and any manual steps need to be removed or become extremely stream-lined. Most people don't mind doing multiple manual steps if the deployment to production only happens once every couple of weeks. Hell, it is still easy to do a manual step or two if deployments only happen once or twice a day. But every 45 minutes? That would get old really fast.
At the time of this writing, it is not possible to deploy code Farm Credit Services of America is not ready to deploy to production every 45 minutes. The whole process made a quantum leap over the past year to get us a lot closer than before. The goal of this article is to explain where we were, where we are, and what still needs to be changed.
This article is not a step by step guide on how to get started in DevOps. Nor will it cover what changes need to be made. I don't know your company or its deployment process. I can only speak from my experiences. I am going to venture a guess and assume the deployment process at your company is different than Farm Credit Services of America. You have to start from somewhere. My recommendation is to get a cross-functional team in a room and discuss the current state of your deployment process, identify the manual steps, and discuss how you can streamline or automate those steps. The cross-functional team should include one or two web admins, DBAs, developers, and a leader sponsor.
When I joined Farm Credit Services of America back in February 2013 this was the process to get a change to production.
During emergency steps 1, 2, 3 (changes were made right in the master branch), and 12 were skipped. The fastest code could get to production during this time was about 2ish hours. And that was when it was really pushed. For example, right after a pre-production request was submitted I would walk over to the web admin area and get them to manually deploy the change as soon as possible. This also meant a lot of extra stress, which meant people were less likely to be diligent and occasionally the fix caused another bug. Or the database delta script missed something.
That 2-hour timeframe is only during an emergency. If the process was completely followed it took about a day or so.
Having a deployment take at least 2 hours when really pushing it automatically excludes it from being able to be deployed to production every 45 minutes. It would be a constant state of chaos. Stress levels would be at a max all the time.
In the past four years, a lot of progress has been made in making this more efficient. Now the process to deploy code to production is:
During an emergency, all steps are followed except the attendance of CRC. All automated tests are run. The two hours has been whittled down to 30 minutes in the event of an emergency. That may not seem like a lot, but the confidence level in each deployment has gotten much higher, it is to the point where for smaller releases the operations staff doesn't need to be online or present during deployments. Automating Database Deployments and adopting Octopus Deploy were the biggest reasons for that drop in time.
This is the point where I wish I could tell you we sat down, wrote out of the process as it stood and where we wanted it to go. But that wasn't really the case. The number one thing people wanted to streamline was the CRC process and get rid of the manual meetings. That drove the direction of almost every decision that was made, from tool choice to deployment philosophies. Along the way, we adopted a number of things to help with that, such as build once/deploy anywhere and automated database deployments.
30 minutes is great, but there is still a lot of room for improvement. That 30 minute timeframe is only in the event of an emergency. It is not something that can be done every 45 minutes. There are still a number of manual steps that need to occur. These either need to be streamlined or automated. The CRC meeting is right now the biggest impediment to the whole deployment process.
At the time of this writing, this would be my ideal future state.
That being the ideal future state, it is important to note the time for deployments can only be whittled down so much. No matter what, a build/deployment cycle will take some amount of time. What is an acceptable amount of time is completely up to you and your company. For me, I would like to see that get whittled down to 15-20 minutes which includes some sort of CRC approval. Some of the ideas being thrown about to help this process further are:
This article was only the start for this subsection of DevOps. It seems like there are many common topics every company encounter on this journey.
I have already written a series of articles for Automated Database Deployments, there is not need to rehash that here. At Farm Credit Services of America, we are beginning down the path of DevOps. Most likely we will stumble along the way. My plan in the coming weeks and months is cover the remaining topics in much more detail and cover what succeeded as well as what failed. Unlike my topic of automated database deployments, which was well on its way to being adopted by Farm Credit Services of America when I started writing on it; the topic of DevOps is going to be covered on this site in much more real-time.