Continuous Integration / Continuous Deployment

Problem statement: Considerable manual intervention is required from developers, QA analysts, and DevOps engineers to put new builds of RDS software into development, testing, and production environments. This intervention is not only tedious but can be error-prone.

Modern DevOps practices, particularly CI/CD (Continuous Integration / Continuous Deployment) can streamline these processes.

DevOps proposes this scenario:

1. The development account in our AWS organization is wide open for any developer to use for experimentation and creation of new services
2. The QA account will be used for “staging” and “release” environments, and when needed a “hotfix” environment, for testing of new services
3. The production account will be managed solely by the DevOps team, and will host our production services and other resources
4. The DevOps account will host, where appropriate, Docker images and other artifacts that are to be shared among the other accounts

We propose that CI/CD pipelines be implemented using GitHub actions that will automatically deploy new versions of services to the “staging” environment when code branches are merged to master in the mammoth repository. This will keep “staging” up-to-date with everything that has passed code review.

The “release” environment will represent the next release that is in preparation, and GitHub actions will keep it up to date with all code that gets merged to the newest release/k2 branch in the mammoth repository. Once the branch is completely assembled for the next release, “release” will be in a state to be used for regression testing (both manual and automated).

Once the new release is fully tested and passed by the QA team, GitHub actions accessible only to the DevOps team will permit automatic replication of the release environment to the production environment.

The GitHub actions that deploy merged code to the staging, release, and production environments will also apply database evolutions to the databases that correspond to those environments. For staging and release, if a commit that includes an evolve is reverted from the branch, the associated devolve will be run.

If a hotfix is needed to a production release, a branch will be made from the previous release/k2 branch and the fixes will be merged to it, bringing the release environment up to date with the hotfix for testing (and eventual promotion to production). Any fixes to the release branch will also need to be merged back to master (and therefore be incorporated into the staging environment).