Skip to main content

Kustomize

Kustomize

Kubernetes native configuration management

Kustomize is used extensively in the Control Plane for customizing Kubernetes manifests across different environments and deployment scenarios, particularly for operators built with Kubebuilder.

Configuration management

Kustomize allows for template-free customization of Kubernetes manifests, enabling consistent deployment of Control Plane operators across different environments.

Overview​

Kustomize is a Kubernetes-native configuration management tool that lets you customize untemplated YAML files for multiple environments while keeping the original files intact. It's now integrated into kubectl and is used extensively in the Control Plane for managing operator deployments.

Kustomize Logo

🔄 Overlay-Based Customization

Modify resources without templating using overlays and patches.

📦 Resource Composition

Combine and organize related resources into cohesive packages.

🎯 Environment-Specific Variants

Create environment-specific variations from a common base.

🔍 No Templates

Use declarative approach without the complexity of templates.

🧩 Component Reuse

Create reusable components for consistent application across deployments.

⚙️ ConfigMap/Secret Generation

Generate ConfigMaps and Secrets from files or literals.

Integration with Kubebuilder​

Kubebuilder and Kustomize

Kubebuilder generates Kustomize-based deployment configurations for all operators in the Control Plane, making it easy to customize deployments for different environments.

Kubebuilder automatically generates a Kustomize-based directory structure for deploying operators:

config/
├── default/
│   ├── kustomization.yaml         # Combines all components
│   ├── manager_auth_proxy_patch.yaml
│   └── manager_config_patch.yaml
├── manager/
│   ├── kustomization.yaml         # Base manager configuration
│   └── manager.yaml               # Controller manager deployment
├── rbac/
│   ├── kustomization.yaml         # Role-based access control
│   ├── leader_election_role.yaml
│   ├── role.yaml
│   └── role_binding.yaml
├── crd/
│   ├── kustomization.yaml         # Custom Resource Definitions
│   └── bases/
│       └── storage.cp.ei.telekom.de_filemanagers.yaml
└── overlays/
    ├── development/
    │   └── kustomization.yaml     # Dev-specific customizations
    └── production/
        └── kustomization.yaml     # Production-specific customizations

Common Commands​

# Preview the generated manifests
kubectl kustomize config/default

# Apply the default configuration
kubectl apply -k config/default

# Apply environment-specific configuration
kubectl apply -k config/overlays/production

# Build manifests and pipe to a file
kubectl kustomize config/overlays/production > production-manifests.yaml

Generator Features​

Resource generation

Kustomize can generate ConfigMaps and Secrets from various sources.

ConfigMap Generation

Generate ConfigMaps from files:

configMapGenerator:
- name: file-manager-config
  files:
  - config.json
  - settings.yaml

This automatically hashes the ConfigMap name for change detection.

Secret Generation

Generate Secrets from literals or files:

secretGenerator:
- name: file-manager-secrets
  type: Opaque
  literals:
  - API_KEY=secret-api-key
  files:
  - tls.crt=certs/tls.crt
  - tls.key=certs/tls.key

Advanced Features​

Variants Management

Manage multiple variants with components:

# kustomization.yaml
components:
- ../components/monitoring
- ../components/high-availability

Components allow reusable sets of patches and resources to be applied across bases.

Resource Transformers

Apply transformations across resources:

transformers:
- metadataTransformer.yaml

# metadataTransformer.yaml
apiVersion: builtin
kind: LabelTransformer
metadata:
  name: addLabels
labels:
  environment: production
  region: eu-central
fieldSpecs:
- path: metadata/labels
  create: true

Patching with Variables

Use variables to make patches more dynamic:

vars:
- name: SERVICE_NAME
  objref:
    kind: Service
    name: file-manager
    apiVersion: v1
  fieldref:
    fieldpath: metadata.name

replacements:
- source:
    kind: Service
    version: v1
    fieldPath: spec.clusterIP
  targets:
  - select:
      kind: Deployment
    fieldPaths:
    - spec.template.spec.containers.[name=app].env.[name=SERVICE_IP].value

Composable Configurations

Create composable, reusable configurations:

# monitoring/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Component

resources:
- prometheus-servicemonitor.yaml

patchesStrategicMerge:
- enable-metrics.yaml

Best Practices​

Kustomize best practices

The Control Plane follows these best practices for Kustomize configurations.

🔄 Clear Base/Overlay Structure

Maintain a clear separation between base resources and environment-specific overlays.

🧩 Modular Components

Create reusable components for features like monitoring, high availability, and security.

📄 Minimal Patching

Keep patches small and focused on specific changes to improve maintainability.

🏷️ Consistent Labels

Use commonLabels to ensure consistent labeling across all resources.

📋 Resource Organization

Group related resources in separate directories (RBAC, CRDs, webhooks).

🧪 Validate Generated Output

Preview and validate manifests before applying them to a cluster.

Common Patterns in Control Plane​

Multi-Environment Deployments

Structure for development, staging, and production environments:

  • Base with shared configuration
  • Environment overlays with specific settings
  • Separate namespaces for isolation
  • Environment-specific resource limits

Operator Customization

Customizing Kubebuilder-generated operators:

  • Adding metrics configuration
  • Configuring webhooks
  • Setting up leader election
  • Adding sidecar containers (e.g., auth proxy)

Versioned Deployments

Managing version updates:

  • Image tag updates for new versions
  • CRD version upgrades
  • Migration strategies between versions
  • Rollback support

Security Configuration

Security-focused customizations:

  • Pod security context configuration
  • RBAC role refinement
  • Network policy application
  • Secret management integration

Comparison with Helm​

Kustomize vs Helm

The Control Plane uses both Kustomize and Helm for different purposes.

Kustomize in Control Plane

When we use Kustomize:

  • Operator deployments (from Kubebuilder)
  • Core infrastructure components
  • GitOps-based deployments
  • CRD management
  • RBAC configuration

Helm in Control Plane

When we use Helm:

  • API server deployments
  • Third-party components

Kubebuilder

Learn about the framework used to build Control Plane operators.

View Kubebuilder

Helm Charts

Explore Helm, the other packaging system used in the Control Plane.

View Helm Charts

Kubebuilder Testing

Learn about testing operators built with Kubebuilder.

View Kubebuilder Testing