/home /blog 29 Jul 2023 | Get ipynb

Git Devops

Motivation

After we have built a web service, the following actions quickly become necesary:

How to deploy it?

Common approaches are to simply dump it into heroku / vercel / any other cloud service or introduce a heavyweight like Ansible / Kubernetes.

A lot of my projects run on cheap VMs on linode. None of these approaches are viable for me without spending a lot of effort maintaining the infrastructure around these.

How to re-deploy on git push / tag?

Integrating tightly with your git host is the common approach. To have netlify / vercel / render.com auto-deploy from github or gitlab.

I can't use that when I don't have an app that fits into the way they want to deploy apps. For example, I want to use a rust and python web server in the same project. Nope. None of the existing system will allow for that directly without having to fiddle around with their settings, at which point I'm investing into their infrastructure.

How to manage separate environments / services?

Besides running on dev machines projects need to run on test/prod/staging/uat and many more environments. Sometimes even on-premises. Managing that is not a trivial task.

Managing secrets / configuration

Cross language projects need to pick up things from configuration. That causes another headache since now you need to manage who has access to what config secrets.

Solution

This is how I manage my projects now. A git dev ops workflow.

Machine setup

  1. All machines need to have git and docker compose installed.
  2. We need to add the following snippet to ~/.bashrc
dk (){
    REPOROOT=$(git rev-parse --show-toplevel)
    ([ ! -z "$REPOROOT" ] && source $REPOROOT/cicd/bashrc 2> /dev/null && $@)
}

Project setup

  1. Single docker-compose.yml file per project in the root of the project
  2. cicd folder to contain scripts we can use. This contains bashrc script that provides management commands for the project.
  3. secrets folder to contain secrets using SOPS. Each environment is in a separate file. We never commit <env>.key files and only commit <env>.enc files.

Example

Base layout

.
├── cicd
│   └── bashrc
├── docker-compose.yml
└── secrets
    ├── ci.enc
    └── ci.key

Environment vars

secrets/bin/set_env.sh is used to set environment variables in CI / prod / dev etc.

#!/usr/bin/env bash

export BIN=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
export SECRETS=$(echo "$BIN/..")
export NAME=$1
export SOPS_AGE_KEY_FILE=$SECRETS/$NAME.key 
export $($BIN/sops --decrypt --input-type dotenv --output-type dotenv $SECRETS/$NAME.enc | xargs)

secrets/bin/edit_env.sh is used to edit env vars using some text editor.

#!/usr/bin/env bash

set -o errexit
set -o pipefail

main (){
    NAME=$1
    BIN=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
    SECRETS=$(echo "$BIN/..")
    KEY_FILE=$(echo "$SECRETS/$NAME.key")
    ENC_FILE=$(echo "$SECRETS/$NAME.enc")
    PLAINTEXT_FILE=$(echo "$SECRETS/$NAME.plaintext")
    echo $BIN
    echo $SECRETS
    echo $KEY_FILE
    echo $ENC_FILE
    echo $PLAINTEXT_FILE
    if [[ -f "$SECRETS/$NAME.enc" ]]; then
        SOPS_AGE_KEY_FILE=$KEY_FILE $BIN/sops --decrypt --input-type dotenv --output-type dotenv $ENC_FILE > $PLAINTEXT_FILE
    fi
    vim $PLAINTEXT_FILE
    $BIN/sops --input-type dotenv --output-type dotenv --encrypt --age $($BIN/age-keygen -y $KEY_FILE) $PLAINTEXT_FILE > $ENC_FILE
    rm $PLAINTEXT_FILE
}
(main $1)

docker-compose.yml

Services

Running project

cicd/bashrc has a few command to allow me to run operations on the project.

reporoot(){
    git rev-parse --show-toplevel
}
set_env_vars() {
    USER_ID=$(id -u)
    echo "
        touch .$ENV.env \
        && source secrets/bin/set_env.sh $ENV \
        && bash secrets/bin/create_envfile.sh $ENV \
        && export USER_ID='$USER_ID' "

}
up(){
    ENV=$1
    PROFILE="${2:-$ENV}"
    cd $(reporoot)
    echo "
    (
        $(set_env_vars) \
        && docker compose --profile $PROFILE \
        up --build --force-recreate -d
    )"
}
down(){
    ENV=$1
    PROFILE="${2:-$ENV}"
    cd $(reporoot)
    echo "
    (
        $(set_env_vars) \
        && docker compose --profile $PROFILE \
        down
    )"
}

This allows me to run commands like: