Secrets

Three-tier secrets management endpoints

The K8s Scheduler implements a three-tier secrets management system using HashiCorp Vault:

1. Organization Secrets - Shared across all deployments in an organization
2. Deployment Secrets - Scoped to a specific deployment
3. Service Secrets - Scoped to a specific service within a deployment

Secrets are stored in Vault and automatically injected into deployments via the init-container pattern.

Deployment-Scoped Secrets

List Secrets

GET /api/secrets

List all secrets for the authenticated user, optionally filtered by deployment.

Authentication: Required (session or API key with secrets:read scope)

Query Parameters:

deployment · string

Filter by deployment name

Response:

id · string

Secret UUID

key · string

Secret key (environment variable name)

deploymentName · string

Deployment name (empty for user-wide secrets)

serviceName · string

Service name (empty for deployment-wide secrets)

description · string

Human-readable description

Example:

curl -X GET "https://your-domain.com/api/secrets?deployment=abc123xyz7" \
  -H "Authorization: Bearer YOUR_API_KEY"

[
  {
    "id": "secret_abc123",
    "key": "DATABASE_PASSWORD",
    "deploymentName": "abc123xyz7",
    "serviceName": "postgres",
    "description": "PostgreSQL admin password"
  },
  {
    "id": "secret_def456",
    "key": "API_KEY",
    "deploymentName": "abc123xyz7",
    "serviceName": "",
    "description": "External API key"
  }
]

Secret values are never returned by the API for security reasons. Only keys and metadata are returned.

---

Create Secret

POST /api/secrets

Create a new secret at deployment or service scope.

Authentication: Required (secrets:write scope)

Request Body:

key · string · required

Secret key (must be valid environment variable name: [A-Za-z_][A-Za-z0-9_]*)

value · string · required

Secret value (stored encrypted in Vault)

deploymentName · string

Deployment name for deployment-scoped secret. Can use template: prefix for template-scoped secrets (e.g., template:nginx)

serviceName · string

Service name for service-scoped secret

description · string

Human-readable description

Response: 201 Created

id · string

Secret UUID

key · string

Secret key

deploymentName · string

Deployment name

serviceName · string

Service name

description · string

Description

Example:

curl -X POST "https://your-domain.com/api/secrets" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "key": "DATABASE_PASSWORD",
    "value": "super-secret-password",
    "deploymentName": "abc123xyz7",
    "serviceName": "postgres",
    "description": "PostgreSQL admin password"
  }'

Validation:

• Key must match pattern: ^[A-Za-z_][A-Za-z0-9_]*$
• Value cannot be empty
• If deploymentName is specified, user must own the deployment
• If deploymentName starts with template:, the template must exist

---

Organization Secrets

Organization secrets are shared across all deployments within an organization. Managing org secrets requires org_owner or org_admin role.

List Organization Secrets

GET /api/org/secrets

List all secrets for the current organization (determined by active team context).

Authentication: Required (must be org owner or admin)

Response:

id · string

Secret UUID

key · string

Secret key

description · string

Description

Example:

curl -X GET "https://your-domain.com/api/org/secrets" \
  -H "Authorization: Bearer YOUR_API_KEY"

[
  {
    "id": "secret_org_123",
    "key": "STRIPE_API_KEY",
    "description": "Stripe payment processing key"
  },
  {
    "id": "secret_org_456",
    "key": "SENDGRID_API_KEY",
    "description": "SendGrid email API key"
  }
]

---

Create Organization Secret

POST /api/org/secrets

Create a new organization-wide secret.

Authentication: Required (org owner or admin)

Request Body:

key · string · required

Secret key (must be valid environment variable name)

value · string · required

Secret value

description · string

Description

Response: 201 Created

Example:

curl -X POST "https://your-domain.com/api/org/secrets" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "key": "STRIPE_API_KEY",
    "value": "sk_live_abc123xyz",
    "description": "Stripe payment processing key"
  }'

---

Update Organization Secret

PUT /api/org/secrets

Update an existing organization secret’s value and/or description.

Authentication: Required (org owner or admin)

Request Body:

id · string · required

Secret UUID

value · string · required

New secret value

description · string

New description

Response: 200 OK

{
  "status": "updated"
}

Example:

curl -X PUT "https://your-domain.com/api/org/secrets" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "secret_org_123",
    "value": "sk_live_new_key_789",
    "description": "Updated Stripe key"
  }'

---

Delete Organization Secret

DELETE /api/org/secrets

Delete an organization secret.

Authentication: Required (org owner or admin)

Request Body:

id · string · required

Secret UUID

Response: 200 OK

{
  "status": "deleted"
}

Example:

curl -X DELETE "https://your-domain.com/api/org/secrets" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"id": "secret_org_123"}'

---

Secrets Hierarchy

Secrets are loaded in the following order (later values override earlier ones):

1. Organization Secrets - Applied to all deployments in the org
2. Deployment Secrets - Applied to all services in the deployment
3. Service Secrets - Applied only to the specific service

This allows for flexible secret management:

• Shared credentials (database URLs, API keys) at org level
• Deployment-specific config at deployment level
• Service-specific secrets at service level

Example Hierarchy

# Organization level (all deployments)
STRIPE_API_KEY=sk_live_abc123
SENDGRID_API_KEY=SG.xyz789

# Deployment level (all services in deployment abc123xyz7)
DATABASE_URL=postgres://host/db
REDIS_URL=redis://host:6379

# Service level (only postgres service)
POSTGRES_PASSWORD=secret123
POSTGRES_USER=admin

---

Vault Storage Paths

Secrets are stored in Vault using the following path structure:

# Organization secrets
orgs/{orgId}/secrets/{key}

# Deployment secrets
deployments/{deploymentName}/secrets/{key}

# Service secrets
deployments/{deploymentName}/services/{serviceName}/secrets/{key}

# Template secrets (copied to deployments on creation)
templates/{templateName}/secrets/{key}
templates/{templateName}/services/{serviceName}/secrets/{key}

---

Secret Injection

Secrets are injected into deployments using an init container pattern:

1. Init Container - Fetches secrets from Vault using deployment’s service account token
2. Shared Volume - Writes secrets to a shared emptyDir volume
3. Main Container - Reads secrets as environment variables on startup

This approach:

• Avoids storing secrets in Kubernetes Secrets
• Provides audit trail in Vault
• Supports dynamic secret rotation
• Works with any Vault authentication method

---

Template Secrets

Templates can define required secrets, which are then copied to each deployment:

services:
  - name: postgres
    image: postgres:15
    secrets:
      - key: POSTGRES_PASSWORD
        description: "Database admin password"
      - key: POSTGRES_USER
        description: "Database admin user"

When creating a deployment from this template:

1. Template secrets are copied to deployments/{deploymentName}/secrets/
2. Service-specific secrets are copied to deployments/{deploymentName}/services/{serviceName}/secrets/
3. Original template secrets remain unchanged

This allows:

• Templates to define required secrets
• Each deployment to have unique secret values
• Secrets to be updated per-deployment without affecting other deployments

---

Security Considerations

Authentication

• Deployments authenticate to Vault using Kubernetes service account tokens
• The operator configures Kubernetes auth backend in Vault
• Each deployment has a unique service account with minimal permissions

Authorization

• Organization secrets require org_owner or org_admin role
• Deployment secrets require deployment ownership
• Service secrets require deployment ownership
• Template secrets require template ownership

Encryption

• All secrets are encrypted at rest in Vault
• Secrets are transmitted over TLS
• Init container runs as non-root user
• Secret files have 0600 permissions

Audit

• All secret operations are logged to the audit log
• Vault maintains its own audit log
• Secret access is tracked per-deployment

---

Backend Configuration

The secrets store backend is configured at server startup:

Vault Backend (Production)

vault:
  address: https://vault.example.com
  token: ${VAULT_TOKEN}
  namespace: "shipyard"

In-Memory Backend (Development)

secrets:
  backend: memory

The in-memory backend is for development only and does not persist secrets across restarts.

---

Error Responses

error · string

Human-readable error message

Common Error Codes

• 400 Bad Request - Invalid key format or missing required field
• 403 Forbidden - Deployment not found, insufficient permissions, or org admin required
• 404 Not Found - Secret not found
• 409 Conflict - Secret with this key already exists at this scope
• 500 Internal Server Error - Vault communication error
• 503 Service Unavailable - Secrets management not configured

Key Validation

Secret keys must match the environment variable name pattern:

• Start with letter or underscore: [A-Za-z_]
• Followed by letters, numbers, or underscores: [A-Za-z0-9_]*

Examples:

• ✅ DATABASE_URL
• ✅ API_KEY
• ✅ _PRIVATE_VAR
• ❌ 1PASSWORD (starts with number)
• ❌ MY-SECRET (contains hyphen)
• ❌ my.secret (contains dot)