How to use HashiCorp Vault and ArgoCD for GitOps


Context #

There is various way to manage your secrets, HashiCorp Vault just happened to be a pretty much widely known approach that I had in mind. So for Humble project, I used it as the core secret management system that go along with Kubernetes secrets themselves. So in this post I'd like to share how I use HashipCorp Vault with ArgoCD to deploy secrets across my system.

For my own convenience: Vault = HashiCorp Vault

Injecting Vault secrets into Kubernetes pods via sidecar #

I followed through the official article. The idea is enable Kubernetes authentication in Vault, bind a Kubernetes Service Account to a role, then setting that role to allow the pods that go with the Service Account to read the secrets in a scoped manner.

Started installing it with the official Helm chart for Vault via Terraform:

The dependency here will be longhorn for the persitent block storage that will be Vault backend:

depends_on       = [helm_release.longhorn]

Remember to switch the injector support on in the values.yaml:

enabled: true

Vault will be up and running in a few seconds, we will initialize and unseal it first, step by step:

kubectl exec -n vault -ti vault-0 /bin/sh
vault operator init
vault operator unseal

Now we define the app policy, which better be provisioned via Terraform as well:

resource "vault_policy" "postgresql_read_only" {
depends_on = [helm_release.vault]
name = "postgresql_read_only"
policy = <<EOT
path "secret/postgresql/*" {
capabilities = ["read"]

First, enable the Kubernets authentication

vault auth enable kubernetes

Now if you were on one of the control plane node of your cluster, the quickest way to set this up is:

vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/" \
kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \

vault write auth/kubernetes/role/myapp \
bound_service_account_names=app \
bound_service_account_namespaces=demo \
policies=app \

Otherwise, you will have to get the JWT token and the ca.crt from one of your Service Account:

export VAULT_SA_NAME=$(kubectl get sa vault-auth) \
--output jsonpath="{.secrets[*]['name']}")
export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME \
--output 'go-template={{.data.token}}' | base64 --decode)
export SA_CA_CRT=$(kubectl config view --raw --minify --flatten \
--output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
export K8S_HOST=$(kubectl config view --raw --minify --flatten \
--output 'jsonpath={.clusters[].cluster.server}')

vault write auth/kubernetes/config \
token_reviewer_jwt="$SA_JWT_TOKEN" \
kubernetes_host="$K8S_HOST" \

Let's create a role and bind that role to our Service Account:

vault write auth/kubernetes/role/postgresql_read_only \
bound_service_account_names=apps \
bound_service_account_namespaces=apps \
policies=postgresql_read_only \

Now let's put some secrets in there!

vault kv put secret/postgresql/data \
username='databaseuser' \
password='suP3rsec(et!' \

Now we can inject it in our pods, refer to the document here

As an example, in my ArgoCD apps folder, I injected the secrets via the ENV VAR "DSN" for Ory Kratos to take it as the connection string to the PostgreSQL database: ory-kratos.yaml

annotations: "true" 'secret/postgresql/data' "postgresql_read_only" |
{{`{{ with secret &quot;secret/postgresql/data&quot; -}}
export DSN="postgres://{{}}:{{}}@postgresql:5432/{{}}?sslmode=disable&max_conns=20&max_idle_conns=4"
{{- end }}`}}

And let the sidecar do it's job!

Using vault-argocd-plugin #

Injecting the secrets into pods is quite easy and straightforward. But what if we wanted to inject it on the fly to other resources like Kubernetes secrets or custom resources? And this is when vault-argocd-plugin comes to aid us.

vault-argocd-plugin is an ArgoCD plugin provided by IBM for helping us doing so. To set this up, first install the executable binary via the initContainer, I used the Helm chart which includes the YAML code for that:

enabled: true
enabled: true
tag: v2.0.0

- name: custom-tools
emptyDir: {}

- name: download-tools
image: alpine:3.8
command: [sh, -c]
- >-
wget -O argocd-vault-plugin &&
chmod +x argocd-vault-plugin &&
mv argocd-vault-plugin /custom-tools/
- mountPath: /custom-tools
name: custom-tools
- name: custom-tools
mountPath: /usr/local/bin/argocd-vault-plugin
subPath: argocd-vault-plugin

Next, we configure plugin so ArgoCD would be able to know how to use it:

configManagementPlugins: |-
- name: argocd-vault-plugin
command: ["argocd-vault-plugin"]
args: ["generate", "./"]
- name: argocd-vault-plugin-helm
command: [sh, -c]
args: ["helm dependency build"]
command: ["sh", "-c"]
args: ["helm template $ARGOCD_APP_NAME . | argocd-vault-plugin generate -"]

Here is the values.yaml file: argocd.yaml

Let's put a new secret

vault kv put secret/humble/demo \
username='locmai' \

Now, create a policy and a role:

kubectl exec -ti vault-0 /bin/sh

cat <<EOF > /home/vault/humble-policy.hcl
path "secret/humble/*" {
capabilities = ["read"]

vault policy write humble /home/vault/humble-policy.hcl

vault write auth/kubernetes/role/myhumbledemo \
bound_service_account_names=default \
bound_service_account_namespaces=argocd \
policies= humble\

Now we could put it all together nice and easy with ArgoCD, let's define a Kubernetes secret manifest that we would like to inject the secret value into:

# demo/secrets.yaml
kind: Secret
apiVersion: v1
name: humble-example-secret
namespace: argocd
annotations: "secret/humble/data/demo"
type: Opaque
username: <username>

The above spec define that we would want to read the secret/humble/demo path (/data/ is the new path pattern for kv-v2 engine) and the secret with the key username will be injected in to placeholder.

Now from the ArgoCD application, we could simply define it to use the plugin as:

path: 'demo'
repoURL: [email protected]:locmai/humble.git
targetRevision: main
name: argocd-vault-plugin
- name: VAULT_ADDR
value: http://vault-ui.vault.svc.cluster.local:8200
- name: AVP_TYPE
value: vault
value: k8s
- name: AVP_K8S_ROLE
value: argocd-server

And the secret injected nice and easy!

Secret injected

Here's some other options people are doing GitOps secrets:

And that's it! There is other ways as well, will post about them when I had the time to try.