Kubernetes Secrets with OCI Vault - External Secrets Operator

This cluster uses OCI Vault (Always Free tier) to securely store all sensitive configuration. This approach ensures that secrets are never committed to the repository and can be retrieved for cluster recreation.

OCI Always Free Limits

Resource	Free Limit	Our Usage
Vault Secrets	150	~10
Master Key Versions	20	1
HSM-protected Keys	✓	✓

Secrets Inventory

All secrets are stored in the oke-secrets-vault OCI Vault:

Secret Name	Purpose	Used By
cloudflare-api-token	DNS automation	External DNS, Cert Manager
cloudflare-zone-id	Cloudflare zone	External DNS
domain-name	Base domain	Manifests
github-pat	Repository access	ArgoCD, GHCR
github-username	GitHub authentication	ArgoCD, GHCR
git-repo-url	Repository URL	ArgoCD
acme-email	Let’s Encrypt contact	Cert Manager
argocd-admin-password	ArgoCD UI login	ArgoCD
argocd-admin-password-hash	Bcrypt hash for argocd-secret	ArgoCD
gemma-api-key	Gemma LLM Authentication	Gemma (scaled to 0)
huggingface-token	HuggingFace model downloads	Gemma (scaled to 0)
gemini-api-key	Google Gemini API key	OpenClaw
openclaw-gateway-token	Web UI auth token	OpenClaw
telegram-bot-token	Telegram Bot API token	OpenClaw
google-places-api-key	Google Places API key	OpenClaw
discord-bot-token	Discord bot token (optional)	OpenClaw
gog-keyring-password	Google Workspace CLI keyring (legacy)	OpenClaw
alphavantage-api-key	Alpha Vantage financial data API key	OpenClaw
bw-client-id	Bitwarden CLI client ID	OpenClaw
bw-client-secret	Bitwarden CLI client secret	OpenClaw
oidc-client-id	OIDC authentication	Open WebUI
oidc-client-secret	OIDC authentication	Open WebUI
oidc-provider-url	OIDC provider endpoint	Open WebUI
ssh-public-key	Instance access (Vault only, not synced to K8s)	OCI Compute

Retrieving Secrets

Prerequisites

Install and configure the OCI CLI:

brew install oci-cli
oci setup config

List All Secrets

oci vault secret list \
  --compartment-id <your-compartment-ocid> \
  --query 'data[].{"name":"secret-name","id":id}' \
  --output table

Retrieve a Secret Value

oci secrets secret-bundle get \
  --secret-id <secret-ocid> \
  --query 'data."secret-bundle-content".content' \
  --raw-output | base64 -d

Retrieve ArgoCD Password

The ArgoCD admin password is automatically synced from Vault to the cluster via External Secrets Operator.

# From cluster (password is synced from Vault)
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d

# Or directly from Vault
ARGOCD_SECRET_ID=$(terraform -chdir=tf-oke output -json secret_ocids | jq -r '.argocd_admin_password')

oci secrets secret-bundle get \
  --secret-id "$ARGOCD_SECRET_ID" \
  --query 'data."secret-bundle-content".content' \
  --raw-output | base64 -d

Generating terraform.tfvars

To recreate terraform.tfvars from OCI Vault:

1. Get all secret OCIDs

   terraform -chdir=tf-oke output -json secret_ocids

2. Create the tfvars file

   cat > tf-oke/terraform.tfvars << 'EOF'
   tenancy_ocid     = "<from OCI Console>"
   user_ocid        = "<from OCI Console>"
   fingerprint      = "<from OCI Console>"
   private_key_path = "/path/to/oci_api_key.pem"
   region           = "<your-region>"
   compartment_ocid = "<from OCI Console>"
   EOF

3. Add secrets from Vault

   # Example for one secret
   echo "cloudflare_api_token = \"$(oci secrets secret-bundle get \
     --secret-id <cloudflare-api-token-ocid> \
     --query 'data."secret-bundle-content".content' \
     --raw-output | base64 -d)\"" >> tf-oke/terraform.tfvars

Tip

The OCI authentication values (tenancy_ocid, user_ocid, fingerprint, private_key_path, region) are not stored in Vault as they are needed to access Vault itself. Store these securely in a password manager.

External Secrets Operator

The cluster runs External Secrets Operator (ESO) to sync OCI Vault secrets to Kubernetes Secrets. This enables GitOps-friendly secret management where:

1. Secrets are stored in OCI Vault
2. ESO reads secrets from Vault
3. ESO creates/updates Kubernetes Secrets
4. Applications reference standard Kubernetes Secrets

Authentication

The cluster uses Instance Principals to authenticate with OCI Vault. This eliminates the need for managing long-lived API keys for the cluster itself.

1. Dynamic Group: oke-nodes-dg groups all instances in the compartment (instance.compartment.id = '<compartment_ocid>').
2. Policy: oke-secrets-read-policy allows the dynamic group to read secret-family and use vaults in the compartment.
3. ClusterSecretStore: Configured with principalType: InstancePrincipal.

Note

The ESO manifests are generated by Terraform from templates in tf-oke/templates/manifests/managed-secrets/. Sensitive values like the Vault OCID are injected at deploy time and gitignored to prevent accidental commits.

Architecture

flowchart LR
    subgraph OCI["OCI Cloud"]
        Vault[(Vault)]
        IAM[IAM Policy]
    end
    
    subgraph K8s["OKE Cluster"]
        Node[Worker Node]
        ESO[External Secrets<br/>Operator]
        css[ClusterSecretStore]
        es[ExternalSecret]
        Secret[K8s Secret]
    end
    
    Node --"Identity"--> IAM
    IAM --"Allow Read"--> Vault
    Vault --"Secret Bundle"--> ESO
    css --"Config"--> ESO
    es --"Ref"--> ESO
    ESO --"Sync"--> Secret

Security Best Practices

1. Never commit secrets - All .tfvars files are gitignored
2. Use HSM protection - OCI Vault uses HSM-backed master keys
3. Enable versioning - Secret versions are retained for rollback
4. Least privilege - Use scoped API tokens (Cloudflare Zone.DNS only)
5. Rotate regularly - Update secrets and create new versions

Terraform Remote State

Terraform state is stored in OCI Object Storage:

Setting	Value
Bucket	oke-tfstate
Versioning	Enabled
Access	Private (NoPublicAccess)
Encryption	Server-side (default)

Caution

Never delete the tfstate bucket. It contains the mapping between Terraform resources and actual OCI infrastructure.