Skip to main content

Deployment Automation

Openlane uses a comprehensive automation system to manage configuration generation, Helm chart updates, and deployment workflows through Buildkite CI/CD pipelines and advanced configuration management.

Overview​

The automation system bridges the gap between application configuration changes in the core repository and infrastructure deployment in the openlane-infra repository through intelligent automation that:

  • Auto-generates Helm values and secret management resources from Go configuration
  • Creates draft PRs showing configuration impact before core changes are merged
  • Manages secrets through External Secrets Operator and GCP Secret Manager
  • Updates Helm charts automatically with proper versioning and changelogs
  • Provides visibility through Slack notifications and linked PR workflows

Buildkite Pipeline Structure​

Core Pipeline (/.buildkite/pipeline.yaml)​

The main pipeline includes several automation-focused steps:

Pre-check Phase​

steps:
  - label: ":yaml: generate config"
    key: "generate_config"
    command: ["task", "config:ci"]

  - label: ":construction: create draft config PR"
    key: "draft_pr_automation"
    depends_on: "generate_config"
    if: build.pull_request.id != null

  - label: ":helm: update helm chart"
    key: "helm_automation"
    depends_on: "generate_config"
    if: build.branch == "main"

Configuration Generation​

The config:ci task runs the schema generator that:

  1. Analyzes Go structs for configuration fields and sensitive data
  2. Generates Helm values with proper schema annotations and comments
  3. Creates External Secrets templates for automatic secret injection
  4. Produces PushSecret resources for manual secret management
  5. Applies domain inheritance for environment-specific deployments

Draft PR Workflow​

For pull requests, the system creates draft PRs in the infrastructure repository:

# Draft PR creation logic
if git diff --name-only | grep -E "(config/|pkg/.*config\.go|jsonschema/)" > /dev/null; then
  create_draft_infrastructure_pr
  link_prs_with_comments
  send_notification_if_needed
fi

Automation Scripts​

Core Scripts​

  • helm-automation.sh - Main Helm chart update automation
  • draft-pr-automation.sh - Creates draft PRs for configuration previews
  • post-merge-pr-automation.sh - Finalizes PRs after core changes merge
  • cleanup-draft-prs.sh - Cleans up stale draft PRs
  • image-tag-automation.sh - Updates image tags for releases

Library Functions​

Located in .buildkite/lib/:

  • common.sh - Shared utilities and logging functions
  • helm.sh - Helm-specific operations and chart management
  • github.sh - GitHub API interactions and PR management
  • templates.sh - Template processing and substitution

Configuration Management System​

Automatic Configuration Generation​

The configuration system uses Go struct reflection to automatically:

Detect Configuration Fields​

type Config struct {
    APIKey string `json:"apiKey" koanf:"apiKey" sensitive:"true"`
    Domain string `json:"domain" koanf:"domain" domain:"inherit"`
    Port   int    `json:"port" koanf:"port" default:"8080"`
}

Generate Helm Values​

# Auto-generated from Go structs
core:
  apiKey: ""  # managed via external secrets
  domain: ""  # inherits from global domain
  port: 8080  # default value from struct

Create Secret Management Resources​

# External Secret template
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: core-apikey
spec:
  secretStoreRef:
    name: gcp-secretstore
    kind: ClusterSecretStore
  target:
    name: core-secrets
    template:
      data:
        CORE_APIKEY: "{{ .apikey }}"
  data:
    - secretKey: apikey
      remoteRef:
        key: core-apikey

Domain Inheritance System​

Global domain configuration automatically populates related fields:

Configuration​

export CORE_DOMAIN=theopenlane.io

Automatic Field Population​

  • SessionDomain → theopenlane.io
  • APIEndpoint → https://api.theopenlane.io (with prefix)
  • WebhookURL → https://api.theopenlane.io/v1/webhook (with prefix and suffix)

Generated Helm Templates​

CORE_SUBSCRIPTION_STRIPEWEBHOOKURL:
{{- if .Values.core.subscription.stripeWebhookURL }}
  {{ .Values.core.subscription.stripeWebhookURL }}
{{- else if .Values.domain }}
  "https://api.{{ .Values.domain }}/v1/stripe/webhook"
{{- else }}
  "https://api.openlane.com/v1/stripe/webhook"
{{- end }}

Secret Management Automation​

External Secrets Integration​

The system automatically detects fields marked with sensitive:"true" and:

  1. Excludes them from plain-text Helm values
  2. Generates External Secret resources for automated injection
  3. Creates PushSecret resources for manual secret management
  4. Configures environment variable mapping

Secret Naming Convention​

For a sensitive field objectStorage.accessKey:

  • Environment Variable: CORE_OBJECTSTORAGE_ACCESSKEY
  • Secret Name: core-objectstorage-accesskey
  • Remote Key: core-objectstorage-accesskey (in GCP Secret Manager)

PushSecret Resources​

Individual files for manual secret management:

# config/pushsecrets/core-objectstorage-accesskey.yaml
apiVersion: v1
kind: Secret
metadata:
  name: core-objectstorage-accesskey
  namespace: openlane-secret-push
data:
  value: <base64-encoded-secret>
---
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: core-objectstorage-accesskey
  namespace: openlane-secret-push
spec:
  secretStoreRef:
    name: gcp-secretstore
    kind: ClusterSecretStore
  selector:
    secret:
      name: core-objectstorage-accesskey
  data:
    - match:
        secretKey: value
        remoteRef:
          remoteKey: core-objectstorage-accesskey

Draft PR Workflow​

Problem Solved​

Before the draft PR system:

  • ❌ Core changes merged without seeing infrastructure impact
  • ❌ Configuration issues discovered during deployment
  • ❌ Bad development loop: merge first, see problems later

Solution​

The draft PR workflow provides full visibility into configuration changes:

1. Developer Opens Core PR​

git checkout -b feature/add-redis-config
# Edit configuration files
git commit -m "Add Redis caching configuration"
git push origin feature/add-redis-config
gh pr create --title "Add Redis caching support"

2. Automatic Draft PR Creation​

  • Detects configuration changes in the core PR
  • Generates new configuration files
  • Creates draft PR in openlane-infra repository
  • Preserves existing Kubernetes settings (replicas, images, etc.)

3. PR Linking​

Both PRs receive comments linking them together:

Core PR Comment:

## 🔗 Related Infrastructure Changes

A draft PR has been automatically created:
**🚧 Draft Infrastructure PR:** https://github.com/theopenlane/openlane-infra/pull/789

### 📋 Review Process
1. Review both PRs together
2. Merge this core PR first
3. Review and merge the infrastructure PR

4. Post-Merge Automation​

After core PR merge:

  • Draft PR automatically converts to ready for review
  • Final configuration changes applied
  • Chart version incremented
  • Slack notification sent to infrastructure team

Configuration Change Detection​

The system triggers draft PR creation for changes to:

Direct Configuration Files:

  • config/helm-values.yaml
  • config/configmap.yaml
  • config/external-secrets/
  • config/pushsecrets/

Configuration Source Files:

  • pkg/objects/config.go
  • internal/ent/entconfig/config.go
  • pkg/entitlements/config.go
  • Schema generator changes

Helm Chart Automation​

Chart Update Process​

When configuration changes are merged to main:

  1. Clone the openlane-infra repository
  2. Apply configuration changes using intelligent merging
  3. Preserve Kubernetes-specific settings (replicas, images, etc.)
  4. Update only core application configuration
  5. Increment chart version automatically
  6. Generate changelog with change summary
  7. Create pull request with detailed description

Intelligent Merging​

Instead of overwriting the entire values.yaml:

# Preserve Kubernetes configuration
yq eval '. as $target | load("generated-core.yaml") as $core |
         $target | .core = $core.core |
         .externalSecrets = $core.externalSecrets' values.yaml

Merged Sections (replaced):

  • core.* - Application configuration
  • externalSecrets.* - Secret management

Preserved Sections (maintained):

  • replicaCount, image.*, service.*
  • ingress.*, resources.*, nodeSelector.*
  • Custom organizational settings

Chart Versioning​

Automatic version increments follow semantic versioning:

# Current version: 1.2.3
# Configuration changes → 1.2.4 (patch increment)
# New features → 1.3.0 (minor increment)
# Breaking changes → 2.0.0 (major increment)

Slack Integration​

Notification Strategy​

  • Silent Drafts: Draft PR creation doesn't send notifications
  • Ready Notifications: Only notify when PRs need review/action
  • Template-Based: External JSON templates for consistency

Webhook Setup​

  1. Create Slack Webhook:

    • Go to Slack workspace settings
    • Navigate to "Incoming webhooks"
    • Create webhook for desired channel
    • Copy webhook URL

  2. Configure Buildkite Secret:

    SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL

Notification Examples​

Helm Chart Update Notification:

{
  "text": "🤖 Helm Chart Update Required",
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*Changes Made:*\n✅ Updated Helm values.yaml\n🔐 Updated External Secrets templates\n📈 Bumped chart version to 1.2.34"
      }
    }
  ]
}

Testing and Validation​

Automated Testing​

Comprehensive test suite validates automation functionality:

# Run all automation tests
./.buildkite/tests/helm-automation.bats

# Test specific components
./.buildkite/tests/draft-pr-automation.bats
./.buildkite/tests/slack-utils.bats

Local Testing​

Test automation scripts locally:

# Test syntax and logic
./.buildkite/test-helm-automation.sh dry-run

# Test with local repository
./.buildkite/test-helm-automation.sh local-repo --verbose

# Test Slack webhooks
SLACK_WEBHOOK_URL=https://hooks.slack.com/... \
  ./.buildkite/test-helm-automation.sh webhook-test

Manual Configuration Testing​

# Regenerate configuration files
go run ./jsonschema/schema_generator.go

# Validate generated Helm values
helm lint ./config/helm-values.yaml

# Test External Secrets templates
kubectl apply --dry-run=client -f config/external-secrets/

Security Considerations​

Secret Handling​

  1. Separation of Concerns: Sensitive values never stored in plain text
  2. Secret Store Integration: All secrets managed through GCP Secret Manager
  3. Individual Secret Control: Each secret can be enabled/disabled independently
  4. Secure Namespacing: PushSecrets use dedicated namespace

Access Control​

  • GitHub Tokens: Limited scope for repository access
  • GCP Service Accounts: Minimal permissions for Secret Manager
  • Buildkite Secrets: Encrypted storage with access controls

Audit Trail​

  • Git History: All configuration changes tracked in version control
  • PR Comments: Detailed change summaries and approval workflows
  • Slack Notifications: Team awareness and audit logging
  • Build Logs: Complete automation execution history

Troubleshooting​

Common Issues​

Draft PR Not Created​

Symptoms: Configuration changes don't trigger draft PR Solutions:

  • Check if configuration files were actually changed
  • Verify GitHub token permissions
  • Review build pipeline logs for errors

Configuration Not Applied​

Symptoms: Changes don't appear in Helm chart Solutions:

  • Ensure config:ci task completed successfully
  • Check for merge conflicts in values.yaml
  • Verify schema generator output

Slack Notifications Missing​

Symptoms: No notifications despite PR creation Solutions:

  • Verify webhook URL configuration
  • Check Slack workspace permissions
  • Test webhook with manual payload

Debugging Commands​

# Check configuration generation
go run ./jsonschema/schema_generator.go

# Validate Helm chart changes
helm template ./charts/openlane --values values.yaml

# Test External Secrets
kubectl apply --dry-run=client -f config/external-secrets/

# Check automation logs
buildkite-agent artifact download "*.log"

Manual Overrides​

Skip Automation​

Add to commit message:

[skip-automation] Manual configuration changes

This change requires manual infrastructure updates.

Force Draft PR​

Add to commit message:

[force-draft-pr] Configuration comment updates

Create draft PR even for minor changes.
Last updated on by Matt Anderson