Vault Module - Shipyard

Overview

The Vault module creates AWS resources required for running HashiCorp Vault on EKS with KMS auto-unseal and DynamoDB as the storage backend. It configures IAM roles for IRSA (IAM Roles for Service Accounts) to allow Vault pods to access AWS services.

Features

KMS Auto-Unseal

AWS KMS key for automatic Vault unsealing on pod restart

DynamoDB Storage

Highly available DynamoDB table for Vault storage backend

IRSA Integration

IAM role for EKS service account with least privilege permissions

Key Rotation

Optional automatic KMS key rotation enabled by default

Point-in-Time Recovery

DynamoDB PITR enabled by default for data protection

Helm-Ready Config

Output configuration object for Vault Helm chart

Architecture

┌────────────────────────────────────────────────────────────┐
│                    EKS Cluster                             │
│                                                             │
│  ┌────────────────────────────────────────────────────┐  │
│  │              Vault Namespace                         │  │
│  │                                                        │  │
│  │  ┌────────────┐  ┌────────────┐  ┌────────────┐  │  │
│  │  │  Vault Pod  │  │  Vault Pod  │  │  Vault Pod  │  │  │
│  │  │  (Active)   │  │ (Standby)  │  │ (Standby)  │  │  │
│  │  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  │  │
│  │         │              │              │         │  │
│  │         └──────────────┼──────────────┘         │  │
│  │                        │                        │  │
│  │         Service Account with IRSA annotation       │  │
│  │         eks.amazonaws.com/role-arn = <IAM_ROLE>     │  │
│  └────────────────────────┬─────────────────────────┘  │
│                         │                               │
└─────────────────────────┼─────────────────────────────────┘
                          │
         ┌────────────────┼────────────────┐
         │                │                │
         ▼                ▼                ▼
┌────────────────┐  ┌───────────────┐  ┌────────────────┐
│   IAM Role    │  │   AWS KMS     │  │   DynamoDB   │
│  (via IRSA)   │  │ (Auto-unseal)│  │  (Storage)   │
└────────────────┘  └───────────────┘  └────────────────┘

Usage Examples

Basic Configuration

module "vault" {
  source = "git@github.com:opsnorth/terraform-modules.git//vault?ref=v1.0.0"

  cluster_name      = "dev-eks-cluster"
  environment       = "dev"
  oidc_provider_arn = module.eks.oidc_provider_arn
  oidc_provider_url = module.eks.oidc_provider_url

  tags = {
    Environment = "dev"
    Team        = "platform"
  }
}

Production Configuration

module "vault" {
  source = "git@github.com:opsnorth/terraform-modules.git//vault?ref=v1.0.0"

  cluster_name      = "prod-eks-cluster"
  environment       = "production"
  oidc_provider_arn = module.eks.oidc_provider_arn
  oidc_provider_url = module.eks.oidc_provider_url

  # Custom namespace and service account
  vault_namespace       = "vault"
  vault_service_account = "vault"

  # Extended KMS key deletion window
  kms_deletion_window_days = 30

  # Enable key rotation
  kms_enable_key_rotation = true

  # Enable DynamoDB point-in-time recovery
  dynamodb_point_in_time_recovery = true

  tags = {
    Environment = "production"
    Critical    = "true"
    Compliance  = "required"
  }
}

With EKS Module

module "eks" {
  source = "git@github.com:opsnorth/terraform-modules.git//eks?ref=v1.0.0"

  cluster_name       = "my-eks-cluster"
  kubernetes_version = "1.34"
  private_subnet_ids = module.vpc.private_subnet_ids
}

module "vault" {
  source = "git@github.com:opsnorth/terraform-modules.git//vault?ref=v1.0.0"

  cluster_name      = module.eks.cluster_id
  environment       = var.environment
  oidc_provider_arn = module.eks.oidc_provider_arn
  oidc_provider_url = module.eks.oidc_provider_url
}

Inputs

Name	Description	Type	Default	Required
`cluster_name`	Name of the EKS cluster	`string`	n/a	yes
`environment`	Environment name	`string`	n/a	yes
`oidc_provider_arn`	ARN of the OIDC provider for EKS	`string`	n/a	yes
`oidc_provider_url`	URL of the OIDC provider for EKS	`string`	n/a	yes
`vault_namespace`	Kubernetes namespace for Vault	`string`	`"vault"`	no
`vault_service_account`	Kubernetes service account for Vault	`string`	`"vault"`	no
`kms_deletion_window_days`	Days before KMS key deletion	`number`	`7`	no
`kms_enable_key_rotation`	Enable KMS key rotation	`bool`	`true`	no
`dynamodb_point_in_time_recovery`	Enable DynamoDB PITR	`bool`	`true`	no
`tags`	Tags to apply to resources	`map(string)`	`{}`	no

Outputs

Name	Description
`kms_key_id`	KMS Key ID for auto-unseal
`kms_key_arn`	KMS Key ARN
`dynamodb_table_name`	DynamoDB table name
`dynamodb_table_arn`	DynamoDB table ARN
`iam_role_arn`	IAM Role ARN for IRSA
`iam_role_name`	IAM Role name
`vault_config`	Configuration object for Helm chart

Deploying Vault with Helm

Use the module outputs to configure Vault Helm chart:

Install Vault

# Add HashiCorp Helm repository
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

# Create namespace
kubectl create namespace vault

# Install Vault
helm install vault hashicorp/vault \
  --namespace vault \
  --values vault-values.yaml

Helm Values (vault-values.yaml)

server:
  ha:
    enabled: true
    replicas: 3
    
    raft:
      enabled: false
    
    config: |
      ui = true
      
      listener "tcp" {
        tls_disable = 1
        address     = "[::]:8200"
        cluster_address = "[::]:8201"
      }
      
      storage "dynamodb" {
        ha_enabled = "true"
        region     = "us-east-1"
        table      = "<DYNAMODB_TABLE_NAME>"
      }
      
      seal "awskms" {
        region     = "us-east-1"
        kms_key_id = "<KMS_KEY_ID>"
      }
      
      service_registration "kubernetes" {}

  serviceAccount:
    create: true
    name: "vault"
    annotations:
      eks.amazonaws.com/role-arn: "<IAM_ROLE_ARN>"

  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "2000m"
      memory: "2Gi"

ui:
  enabled: true
  serviceType: "ClusterIP"

Terraform-Generated Helm Values

resource "helm_release" "vault" {
  name       = "vault"
  repository = "https://helm.releases.hashicorp.com"
  chart      = "vault"
  namespace  = module.vault.vault_config.namespace
  version    = "0.27.0"

  values = [
    yamlencode({
      server = {
        ha = {
          enabled  = true
          replicas = 3
          config   = <<-EOT
            ui = true

            listener "tcp" {
              tls_disable = 1
              address     = "[::]:8200"
              cluster_address = "[::]:8201"
            }

            storage "dynamodb" {
              ha_enabled = "true"
              region     = "${var.aws_region}"
              table      = "${module.vault.dynamodb_table_name}"
            }

            seal "awskms" {
              region     = "${var.aws_region}"
              kms_key_id = "${module.vault.kms_key_id}"
            }

            service_registration "kubernetes" {}
          EOT
        }
        serviceAccount = {
          create = true
          name   = module.vault.vault_config.service_account
          annotations = {
            "eks.amazonaws.com/role-arn" = module.vault.iam_role_arn
          }
        }
        resources = {
          requests = {
            cpu    = "500m"
            memory = "1Gi"
          }
          limits = {
            cpu    = "2000m"
            memory = "2Gi"
          }
        }
      }
      ui = {
        enabled     = true
        serviceType = "ClusterIP"
      }
    })
  ]
}

Initializing Vault

Initialize and Unseal

# Wait for pods to be ready
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=vault -n vault

# Initialize Vault (only run once)
kubectl exec -n vault vault-0 -- vault operator init \
  -key-shares=5 \
  -key-threshold=3 \
  -format=json > vault-keys.json

# Save the keys securely!
cat vault-keys.json | jq -r .root_token
cat vault-keys.json | jq -r .unseal_keys_b64[]

Store the root token and unseal keys in a secure location (e.g., AWS Secrets Manager, 1Password). They cannot be recovered if lost.

Check Vault Status

# Check seal status
kubectl exec -n vault vault-0 -- vault status

# All pods should auto-unseal using KMS
kubectl get pods -n vault

Enable Secrets Engine

# Login with root token
kubectl exec -n vault vault-0 -- vault login <ROOT_TOKEN>

# Enable KV v2 secrets engine
kubectl exec -n vault vault-0 -- vault secrets enable -path=secret kv-v2

# Write a test secret
kubectl exec -n vault vault-0 -- vault kv put secret/test password=secret123

# Read the secret
kubectl exec -n vault vault-0 -- vault kv get secret/test

Kubernetes Authentication

Enable Kubernetes auth for applications:

# Enable Kubernetes auth
kubectl exec -n vault vault-0 -- vault auth enable kubernetes

# Configure Kubernetes auth
kubectl exec -n vault vault-0 -- vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# Create a policy
kubectl exec -n vault vault-0 -- vault policy write app-policy - <<EOF
path "secret/data/app/*" {
  capabilities = ["read", "list"]
}
EOF

# Create a role
kubectl exec -n vault vault-0 -- vault write auth/kubernetes/role/app \
  bound_service_account_names=app \
  bound_service_account_namespaces=default \
  policies=app-policy \
  ttl=24h

External Secrets Operator Integration

Configure ESO to sync secrets from Vault:

ClusterSecretStore

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://vault.vault.svc.cluster.local:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"
          serviceAccountRef:
            name: "external-secrets"
            namespace: "external-secrets-system"

ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
  namespace: default
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: app-secrets
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: secret/data/app/database
      property: password

Backup and Recovery

DynamoDB Backup

Point-in-Time Recovery (PITR):

Enabled by default via dynamodb_point_in_time_recovery = true
Allows restore to any point in the last 35 days

Manual Backup:

aws dynamodb create-backup \
  --table-name <DYNAMODB_TABLE_NAME> \
  --backup-name vault-backup-$(date +%Y%m%d)

KMS Key Backup

KMS keys cannot be exported, but:

Key material is replicated across multiple AZs
Key rotation creates new key versions
Old versions remain available for decryption

Vault Data Export

# Export all secrets (requires root token)
kubectl exec -n vault vault-0 -- vault kv list -format=json secret/ > secrets-list.json

# Export individual secrets
for secret in $(cat secrets-list.json | jq -r '.[]'); do
  kubectl exec -n vault vault-0 -- vault kv get -format=json secret/$secret > $secret.json
done

Cost Considerations

KMS Costs

KMS key: $1/month
API requests: $0.03 per 10,000 requests
Vault unsealing: ~1000 requests/day = ~$0.10/month
Total: ~$1.10/month

DynamoDB Costs

On-demand pricing (pay per request)
Typical Vault usage: ~$5-20/month
PITR: Additional 20% of table storage
Consider provisioned capacity for predictable workloads

EKS Pod Costs

3 Vault pods @ 500m CPU, 1Gi memory
Runs on existing EKS node group (no additional compute cost)
Storage: ~1GB per pod for cache

Troubleshooting

Vault Pods Not Starting

Check IAM role:

kubectl describe pod -n vault vault-0 | grep ServiceAccount
kubectl describe serviceaccount -n vault vault

Verify IRSA annotation:

kubectl get serviceaccount -n vault vault -o yaml | grep role-arn

Check pod logs:

kubectl logs -n vault vault-0

Cannot Auto-Unseal

Test KMS access from pod:

kubectl exec -n vault vault-0 -- aws kms describe-key --key-id <KMS_KEY_ID>

Common issues:

IAM role missing kms:Decrypt permission
KMS key not in the same region as EKS
OIDC provider not configured correctly

DynamoDB Access Issues

Test DynamoDB access:

kubectl exec -n vault vault-0 -- aws dynamodb describe-table --table-name <TABLE_NAME>

Check IAM permissions:

aws iam get-role-policy \
  --role-name <IAM_ROLE_NAME> \
  --policy-name vault-policy

Vault Initialization Failed

Check seal status:

kubectl exec -n vault vault-0 -- vault status

If sealed manually, unseal:

kubectl exec -n vault vault-0 -- vault operator unseal <UNSEAL_KEY_1>
kubectl exec -n vault vault-0 -- vault operator unseal <UNSEAL_KEY_2>
kubectl exec -n vault vault-0 -- vault operator unseal <UNSEAL_KEY_3>

With KMS auto-unseal configured, manual unsealing should not be necessary.

Security Best Practices

Secure Root Token

Store root token in AWS Secrets Manager or a password manager. Revoke after initial setup.

Enable Audit Logging

Configure Vault audit logging to CloudWatch or S3 for compliance.

Restrict Network Access

Use NetworkPolicies to limit Vault access to authorized pods only.

Key Rotation

Enable KMS key rotation (enabled by default) for enhanced security.

Least Privilege IAM

IAM role has minimum permissions for KMS and DynamoDB access only.

Enable PITR

DynamoDB point-in-time recovery (enabled by default) protects against data loss.

EKS Module

Deploy EKS cluster for Vault

K8s Scheduler Secrets

Using Vault for secrets management

Infrastructure Guide

Complete deployment workflow

Dependencies

External Secrets Operator setup

​Overview

​Features

KMS Auto-Unseal

DynamoDB Storage

IRSA Integration

Key Rotation

Point-in-Time Recovery

Helm-Ready Config

​Architecture

​Usage Examples

​Basic Configuration

​Production Configuration

​With EKS Module

​Inputs

​Outputs

​Deploying Vault with Helm

​Install Vault

​Helm Values (vault-values.yaml)

​Terraform-Generated Helm Values

​Initializing Vault

​Initialize and Unseal

​Check Vault Status

​Enable Secrets Engine

​Kubernetes Authentication

​External Secrets Operator Integration

​ClusterSecretStore

​ExternalSecret

​Backup and Recovery

​DynamoDB Backup

​KMS Key Backup

​Vault Data Export

​Cost Considerations

​Troubleshooting

​Vault Pods Not Starting

​Cannot Auto-Unseal

​DynamoDB Access Issues

​Vault Initialization Failed

​Security Best Practices

Secure Root Token

Enable Audit Logging

Restrict Network Access

Key Rotation

Least Privilege IAM

Enable PITR

​Related Documentation

EKS Module

K8s Scheduler Secrets

Infrastructure Guide

Dependencies

Overview

Features

Architecture

Usage Examples

Basic Configuration

Production Configuration

With EKS Module

Inputs

Outputs

Deploying Vault with Helm

Install Vault

Helm Values (vault-values.yaml)

Terraform-Generated Helm Values

Initializing Vault

Initialize and Unseal

Check Vault Status

Enable Secrets Engine

Kubernetes Authentication

External Secrets Operator Integration

ClusterSecretStore

ExternalSecret

Backup and Recovery

DynamoDB Backup

KMS Key Backup

Vault Data Export

Cost Considerations

Troubleshooting

Vault Pods Not Starting

Cannot Auto-Unseal

DynamoDB Access Issues

Vault Initialization Failed

Security Best Practices

Related Documentation