How to use HashiCorp Vault and ArgoCD for GitOps

Context

There are various ways to manage your secrets. HashiCorp Vault (Vault from here on) is a widely known option, and it was the one I had in mind. For the Humble project, I used it as the core secret management system alongside Kubernetes secrets themselves. In this post I’d like to share how I use Vault with ArgoCD to deploy secrets across my system.

Injecting Vault secrets into Kubernetes pods via sidecar

I followed the official article. The idea is to enable Kubernetes authentication in Vault, bind a Kubernetes Service Account to a role, then set that role to allow pods using the Service Account to read the secrets in a scoped manner.

I installed it with the official Helm chart for Vault via Terraform: init-resources.tf.

The dependency here is longhorn for the persistent block storage that backs Vault:

depends_on       = [helm_release.longhorn]

Remember to switch the injector support on in the values.yaml:

injector:
  enabled: true

Vault will be up and running in a few seconds. We initialize and unseal it first, step by step:

kubectl exec -n vault -ti vault-0 /bin/sh
vault operator init
vault operator unseal

Now we define the app policy, which should also be provisioned via Terraform:

resource "vault_policy" "postgresql_read_only" {
  depends_on = [helm_release.vault]
  name       = "postgresql_read_only"
  policy     = <<EOT
  path "secret/postgresql/*" {
    capabilities = ["read"]
}
EOT
}

First, enable the Kubernetes authentication:

vault auth enable kubernetes

If you’re on one of the control plane nodes of your cluster, the quickest way to set this up is:

vault write auth/kubernetes/config \
   token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
   kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
vault write auth/kubernetes/role/myapp \
   bound_service_account_names=app \
   bound_service_account_namespaces=demo \
   policies=app \
   ttl=1h

Otherwise, you will have to get the JWT token and the ca.crt from one of your Service Accounts:

export VAULT_SA_NAME=$(kubectl get sa vault-auth -o jsonpath="{.secrets[*]['name']}")
export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o go-template='{{ .data.token }}' | base64 --decode)
export SA_CA_CRT=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode)
export K8S_HOST=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.server}')
vault write auth/kubernetes/config \
   token_reviewer_jwt="$SA_JWT_TOKEN" \
   kubernetes_host="$K8S_HOST" \
   kubernetes_ca_cert="$SA_CA_CRT"

Let’s create a role and bind it to our Service Account:

vault write auth/kubernetes/role/postgresql_read_only \
   bound_service_account_names=apps \
   bound_service_account_namespaces=apps \
   policies=postgresql_read_only \
   ttl=1h

Now let’s put some secrets in there:

vault kv put secret/postgresql/data \
      username='databaseuser' \
      password='suP3rsec(et!' \
      ttl='30s'

Now we can inject it into our pods. See the documentation at https://www.vaultproject.io/docs/platform/k8s/injector.

As an example, in my ArgoCD apps folder I injected the secret as the DSN env var so Ory Kratos uses it as the connection string to the PostgreSQL database: ory-kratos.yaml.

And let the sidecar do its job.

Using argocd-vault-plugin

Injecting the secrets into pods is straightforward. But what if we want to inject them on the fly into other resources like Kubernetes secrets or custom resources? That is where argocd-vault-plugin helps.

To set this up, first install the executable binary in an initContainer. The Helm chart values include this:

repoServer:
    metrics:
        enabled: true
        serviceMonitor:
            enabled: true
    image:
        tag: v2.0.0
    volumes:
    - name: custom-tools
      emptyDir: {}
    initContainers:
    - name: download-tools
      image: alpine:3.8
      command: [sh, -c]
      args:
        - >-
          wget -O argocd-vault-plugin
          https://github.com/IBM/argocd-vault-plugin/releases/download/v1.1.1/argocd-vault-plugin_1.1.1_linux_amd64 &&
          chmod +x argocd-vault-plugin &&
          mv argocd-vault-plugin /custom-tools/
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
    volumeMounts:
    - name: custom-tools
      mountPath: /usr/local/bin/argocd-vault-plugin
      subPath: argocd-vault-plugin

Next, we configure the plugin so ArgoCD knows how to use it:

server:
  config:
    configManagementPlugins: |-
      - name: argocd-vault-plugin
        generate:
          command: ["argocd-vault-plugin"]
          args: ["generate", "./"]
      - name: argocd-vault-plugin-helm
        init:
          command: [sh, -c]
          args: ["helm dependency build"]
        generate:
          command: ["sh", "-c"]
          args: ["helm template $ARGOCD_APP_NAME . | argocd-vault-plugin generate -"]

Here is the values.yaml file: argocd.yaml.

Let’s put a new secret:

vault kv put secret/humble/demo \
  username='locmai' \
  ttl='30s'

Now create a policy and a role:

kubectl exec -ti vault-0 /bin/sh
cat <<EOF > /home/vault/humble-policy.hcl
path "secret/humble/*" {
  capabilities = ["read"]
}
EOF
vault policy write humble /home/vault/humble-policy.hcl
vault write auth/kubernetes/role/myhumbledemo \
   bound_service_account_names=default \
   bound_service_account_namespaces=argocd \
   policies=humble \
   ttl=1h

Now we can put it all together with ArgoCD. Define a Kubernetes secret manifest that we would like to inject the secret value into:

# demo/secrets.yaml
kind: Secret
apiVersion: v1
metadata:
  name: humble-example-secret
  namespace: argocd
  annotations:
    avp.kubernetes.io/path: "secret/humble/data/demo"
type: Opaque
stringData:
  username: <username>

The spec above tells the plugin to read the secret/humble/demo path (/data/ is the new path pattern for the kv-v2 engine) and inject the secret with the key username into the <username> placeholder.

From the ArgoCD application, we can configure it to use the plugin like this:

spec:
  source:
    path: 'demo'
    repoURL: [email protected]:locmai/humble.git
    targetRevision: main
    plugin:
      name: argocd-vault-plugin
      env:
        - name: VAULT_ADDR
          value: http://vault-ui.vault.svc.cluster.local:8200
        - name: AVP_TYPE
          value: vault
        - name: AVP_AUTH_TYPE
          value: k8s
        - name: AVP_K8S_ROLE
          value: argocd-server

And the secret is injected, nice and easy.

Other GitOps secrets approaches: https://argo-cd.readthedocs.io/en/stable/operator-manual/secret-management/.