More often than not, deployments in the enterprise are scary and high-risk. Often, deployment involves manually executing a complicated procedure that is only tested on deployment day. Binaries have to be moved around, configuration files need to be backed up and modified, services need to be restarted. Each product has a different procedure, depending on which developer or team created the product.
Many organizations never evolve beyond this point. Dev remains content to throw these complicated deployment procedures over the fence for ops to deal with, and ops remains content to sit there and take it. If somebody realizes that there is in fact a problem, the initial reaction is often that of any good technologist when encountering an error-prone, non-repeatable, and highly manual process: shouting “we should automate this”! Perhaps a clever ops guy writes some perl scripts to make some of the procedures simpler. Perhaps they go off and bring in a fancy enterprise task scheduler or a more purpose-specific deployment automation system in the hopes that an off-the-shelf product can solve their deployment problem. The enterprising technologist then goes to a lot of trouble to shoe-horn all the existing procedures into “automated” workflows or scripts.
What happens then? Every new product requires a new script. Application configuration changes probably require many scripts to be modified in subtly different ways. New steps in the deployment procedure will require new steps in the “automated” workflow. The team ends up with an “automated” process that requires a lot of manual maintenance. Automation doesn’t simplify the process, it simply rearranges the deck chairs of complexity.
Attempts to automate away deployment complexity are just a band-aids, they do not address the root problem. While it is a good idea to automate deployment, it is not a good idea to leave the existing complex and disparate deployment procedures in place and simply “automate” them. The better approach is to focus on creating a consistent and simple automated deployment procedure that can be applied to all applications. This saves the trouble of inventing a deployment procedure for every new product. If all products utilize the same simple deployment procedure, and a deployment can be reduced to an easy command line call, perhaps there won’t be a need for that fancy enterprise scheduling system after all.
Here are a few experience-tested principles to follow when attempting to improve the quality of your deployment procedures:
- Solve the configuration problem before trying to solve the deployment problem. Many of the difficulties encountered during deployment relate to configuration — changing the DEV database connection string to the PROD one, ensuring that the correct IP addresses are being used for web services, etc. Any deployment procedure that requires modification of application-specific config files (especially if each application uses a different file format) goes against our goal of simplicity and consistency. A consistent, firm-wide approach to handling application configuration is a prerequisite to a consistent, simple deployment process. Most solutions to the configuration problem will require a centralized configuration service so that configuration can be specified outside of code or config files. This service should basically look like a key-value store where the value can be different based on the environment (and possibly other factors, like application or version). The required configuration values can be injected into config files at deploy-time by the deployment process, or they can be fetched at run-time by a shared library. My preference is for the second approach, because it allows applications to be re-configured without being re-deployed. Local configuration files are a huge impediment to simple deployments, and eliminating them should be your top priority when trying to improve your deployment process.
- Minimize or eliminate environmental dependencies. Ideally, everything an application needs to run should be copied to the application’s install directory during the deployment. Obviously, we have to make exceptions for the operating system and the runtime environment — but ideally those are the only exceptions. Your application should not rely on registered dlls or dlls in the Global Assembly Cache. A “stock” machine at your organization should be able to run your proprietary apps without any additional software installed as a prerequisite. This simplifies the deployment task for ops and reduces risk for your users.
- The deployment procedure should only be one step long. Running a deployment should be a “one click” or “one command” operation. Operations should be able to deploy software by specifying the application to be deployed and the machine on which to deploy it, then click a button (or hit return) and not do anything else. This process should be consistent across all products so that everyone in the organization knows how to deploy all products in the portfolio.
- The deployment procedure should not involve manual modification of files on the target machine. Deploying software should not require operations to manually copy-paste or edit files on the target machine. After running the “one step” described above, operations should not need to move any additional files, create desktop shortcuts, or modify configuration files (which won’t be required if you’ve solved the configuration problem). All of these tasks should be performed in a consistent way by your deployment process.
- Build deployment functionality into shared application frameworks. If the goal is to reduce deployment to a single command-line call, why not make that a call to the application itself? Making applications self-deploying has a number of benefits. It helps ensure that all products will support the correct deployment procedure (since it comes “for free” along with the framework your developers already use) and it simplifies the deployment process by eliminating the need for a separate “deployment runner” application. This approach can also have an organizational benefit because placing deployment logic in the application’s code reinforces the notion that developers should take some responsibility for the deployment procedure.
When embarking upon a project to fix an organization’s deployment woes, it is important to realize that the problems cannot be papered over with automation. Configuration must be standardized and centralized, dependencies must be organized to make a one-step deployment possible, and application frameworks must be modified (or created) to support deployment. Truly solving the deployment problem will require changes to application code, not the addition of a magical deployment automation system. Creating the deployment process is a development concern, not an operational concern.
