Skip to main content

S3 Storage with MinIO

S3 Storage with MinIO

High-performance object storage for the Control Plane's file management

The file-manager component of the Control Plane uses MinIO as an S3-compatible storage backend for file operations.

Object storage for cloud-native applications

MinIO provides a high-performance, S3-compatible object storage system that allows the Control Plane to store and retrieve files efficiently while maintaining compatibility with cloud environments.

Overview​

MinIO is a high-performance, S3-compatible object storage system. The file-manager component implements storage interfaces that use the MinIO Go client library (minio-go/v7) to provide file upload and download capabilities.

MinIO Logo

Why MinIO?​

🔄 S3 Compatibility

Full compatibility with Amazon S3 API, ensuring portability across cloud environments.

⚡ High Performance

Optimized for high throughput and low latency operations, even with large files.

🔒 Built-in Security

Support for TLS encryption, IAM policies, and encryption-at-rest.

ðŸŠķ Lightweight

Small footprint makes it ideal for containerized deployments in Kubernetes.

☁ïļ Cloud Native

Designed for modern cloud environments with distributed architecture.

ðŸ“Ķ Scalability

Scales horizontally to accommodate growing storage needs.

Integration Architecture​

Clean architecture

The layered approach ensures that the file-manager component can be easily extended to support other storage backends in the future by implementing the same interfaces.

Backend Interface​

Interface-Driven Design

The file-manager uses a clean interface-driven design to:

  • Abstract storage implementation details
  • Enable multiple backend implementations
  • Facilitate testing with mock implementations
  • Enforce consistent behavior across backends
  • Allow for future backend additions without API changes

Storage Interfaces

Key interfaces include:

  • FileUploader: For file upload operations
  • FileDownloader: For file retrieval operations
Interface segregation

Following the Interface Segregation Principle, the file-manager splits functionality into focused interfaces rather than a single monolithic one. This allows components to depend only on the functionality they need.

File Identifiers​

Structured file IDs

The file-manager uses a structured approach for file identifiers that incorporates environment, group, team, and filename to ensure proper organization and access control.

File ID Structure Example​

File ID Format

environment--group--team--filename

For example:

  • prod--finance--reporting--quarterly-results.xlsx
  • dev--marketing--campaigns--summer-campaign.pdf
  • test--engineering--platform--load-test-results.json

This structure allows for:

  • Multi-tenancy through environment isolation
  • Organizational hierarchy reflection
  • Team-based access control
  • Logical file organization

Checksum Validation​

Data integrity

Checksum validation is critical for ensuring file integrity throughout the upload and download process. Always verify file checksums when handling user uploads.

Security Considerations​

🔐 Credentials Management

Access keys are managed securely through Kubernetes secrets and environment variables.

🔒 TLS Encryption

Communication with MinIO uses TLS encryption when useSSL is enabled.

ðŸ›Ąïļ JWT Authentication

API endpoints require valid JWT tokens, ensuring only authorized users can access files.

ðŸ‘Ī Access Control

File IDs include group and team information to enable fine-grained access control.

🔍 Audit Logging

All file operations are logged with user context for auditability.

⏱ïļ Temporary URLs

Support for time-limited presigned URLs for secure file sharing.

Configuration​

The S3 backend is configured through environment variables or config files:

# Example configuration
s3:
  endpoint: "s3.example.com"
  accessKey: "${S3_ACCESS_KEY}"
  secretKey: "${S3_SECRET_KEY}"
  bucket: "file-manager"
  useSSL: true

Kubernetes Configuration​

Kubernetes integration

In a Kubernetes environment, the MinIO credentials are typically stored in a Secret and injected as environment variables.

Advanced Features​

Object Lifecycle Management

The file-manager supports object lifecycle policies for:

  • Automatic deletion of temporary files
  • Moving older files to archive storage
  • Setting retention policies by file type
  • Version management of important documents

Multi-part Uploads

For large files, multi-part uploads provide:

  • Better handling of network interruptions
  • Parallel upload of file parts
  • Progress tracking for large transfers
  • Improved performance for large files

File Manager

Learn about the file manager component that uses MinIO for storage.

Component Overview

Go Language

Explore the foundational language used in the Control Plane.

Go Language