Docker Build Guide

Overview

This guide covers the complete Docker containerization strategy for the ShipClojure-Datom application, including multi-stage builds, JVM optimization, and production deployment with Kamal.

Key Objectives:

• Optimize build times with effective layer caching
• Configure JVM for container-aware memory management
• Ensure production observability and debugging capabilities
• Integrate seamlessly with Kamal deployment workflow

Multi-stage Build Process

The Docker build uses a two-stage approach to create an optimized production image.

Builder Stage

The builder stage handles all compilation and asset generation:

• Node.js dependencies and frontend compilation
• Babashka for build automation
• Java class compilation
• Shadow-CLJS for ClojureScript compilation

FROM clojure:tools-deps-1.12.1.1550-bookworm-slim AS builder

# Install Node.js for frontend compilation
RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && \
    apt-get install -y nodejs

# Install Babashka for build automation
RUN curl -sLO https://raw.githubusercontent.com/babashka/babashka/master/install && \
    chmod +x install && ./install

# Copy dependencies first for layer caching
COPY deps.edn ./
COPY package*.json ./
RUN npm ci

# Copy source and build
COPY . .
RUN bb uberjar

Runtime Stage

The runtime stage creates a minimal production image:

• Only runtime dependencies
• Compiled assets from builder stage
• Security hardening with non-root user
• Optimized for container orchestration

FROM eclipse-temurin:21.0.5_11-jre-jammy AS runtime

# Create non-root user for security
RUN groupadd -r clojure && useradd -r -g clojure clojure

# Copy compiled application from builder
COPY --from=builder --chown=clojure:clojure /app/target/app.jar /opt/app/app.jar
COPY --from=builder --chown=clojure:clojure /app/resources /opt/app/resources

# Set working directory and user
WORKDIR /opt/app
USER clojure

# Configure JVM options (see sections below for details)
ENV JDK_JAVA_OPTIONS="${JAVA_MEMORY_OPTS} ${JAVA_GC_OPTS} ${JAVA_DEBUG_OPTS} ${JAVA_APP_OPTS}"

EXPOSE 5050
CMD ["java", "-jar", "app.jar"]

Container Security & JVM Optimization

Security is implemented through multiple layers of protection, while the JVM is specifically tuned for container environments and SaaS applications with real-time features.

Security Configuration

The runtime container runs as a non-privileged user to reduce security risks:

RUN groupadd -r clojure && useradd -r -g clojure clojure
USER clojure

Only essential runtime packages are installed to reduce attack surface. The runtime image uses the minimal JRE instead of the full JDK.

Container-Aware Memory Management

Traditional JVM memory settings like -Xms512m -Xmx2g are problematic in containers because:

• They don't adapt to actual container memory limits
• Can cause OOM kills if container has less memory than expected
• Don't scale when container resources change

Solution: Use percentage-based memory allocation that adapts to container limits.

# Memory management: Container-aware settings that adapt to available RAM
# UseContainerSupport: JVM detects container memory limits instead of host memory
# MaxRAMPercentage=35.0: Use 35% of container memory for heap (conservative for shared server)
# InitialRAMPercentage=15.0: Start with 15% heap to reduce startup memory pressure
ENV JAVA_MEMORY_OPTS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=35.0 -XX:InitialRAMPercentage=15.0"

Memory Allocation Strategy for an 8GB Hetzner server running multiple services:

• App container: ~35% of available memory (~2.8GB heap)
• PostgreSQL: Default PostgreSQL memory management
• Datomic transactor: Configured separately via transactor properties
• Memcached: 512MB cache via command arguments
• System overhead: Remaining memory for OS and Docker

Garbage Collection Optimization

G1 Garbage Collector is optimal for low-latency SaaS applications with real-time features like WebSocket connections.

# Garbage collection: G1GC optimized for low-latency SaaS applications
# UseG1GC: Low-latency collector ideal for real-time WebSocket connections (Sente)
# MaxGCPauseMillis=200: Target 200ms max pause (realistic for container environment)
# G1HeapRegionSize=16m: Optimal region size for medium heaps (~2-3GB)
# InitiatingHeapOccupancyPercent=35: Start concurrent GC early to avoid allocation pressure
# UseStringDeduplication: Save memory from duplicate strings (JWTs, schemas, queries)
ENV JAVA_GC_OPTS="-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:G1HeapRegionSize=16m -XX:InitiatingHeapOccupancyPercent=35 -XX:+UseStringDeduplication"

Why These Settings Matter:

• Real-time WebSockets: Low GC pause times prevent connection drops
• CQRS Commands: Predictable response times for user actions
• String deduplication: Saves memory with JWT tokens, schema definitions, and Datalog queries

Performance & Observability

Additional JVM optimizations for functional Clojure code and production debugging:

# Clojure optimizations: Performance improvements for functional code
# direct-linking=true: Optimize function calls by bypassing var lookups (5-15% speedup)
ENV JAVA_APP_OPTS="-Dclojure.compiler.direct-linking=true"

# Production observability: Essential monitoring for container deployment
# Xlog:gc: Write GC logs to persistent volume for performance analysis
# HeapDumpOnOutOfMemoryError: Capture heap dumps for debugging OOM issues
# HeapDumpPath=/opt/app/logs/: Store dumps in app directory with mounted volume
ENV JAVA_DEBUG_OPTS="-Xlog:gc*:/opt/app/logs/gc-%t.log:time,tags -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/app/logs/"

Benefits for ShipClojure-Datom:

• Optimizes CQRS handler chains
• Improves DataScript query performance
• Perfect for production where functions aren't redefined at runtime
• GC logs track memory allocation patterns and pause times
• Heap dumps enable post-mortem analysis of OutOfMemoryError events
• Persistent volumes ensure data survives container restarts and deployments

Considerations:

• May break REPL-driven development (disable for dev profiles)
• Safe for pure functional architectures like ours

Environment Configuration & Resource Management

JVM configuration is split into logical groups for maintainability and flexibility, while resources are carefully allocated across all services.

Modular JVM Configuration

Split JVM options into logical groups for easier maintenance:

ENV JAVA_MEMORY_OPTS="..."  # Memory management
ENV JAVA_GC_OPTS="..."      # Garbage collection
ENV JAVA_DEBUG_OPTS="..."   # Observability
ENV JAVA_APP_OPTS="..."     # Application-specific

# Combined at build time
ENV JDK_JAVA_OPTIONS="${JAVA_MEMORY_OPTS} ${JAVA_GC_OPTS} ${JAVA_DEBUG_OPTS} ${JAVA_APP_OPTS}"

Environment-specific settings use JAVA_TOOL_OPTIONS in Kamal deployment:

env:
  clear:
    # Runtime-specific Datomic configuration (added to base JVM options)
    # This gets automatically combined with JDK_JAVA_OPTIONS by the JVM
    JAVA_TOOL_OPTIONS: "-Ddatomic.memcachedServers=shipclojure-datom-memcached:11211"

Configuration Benefits:

• No duplication: Base JVM tuning stays in Dockerfile
• Environment flexibility: Override specific option groups in Kamal
• Clean separation: Build-time vs runtime configuration

Resource Management Strategy

Memory allocation strategy across all services on a shared server.

On an 8GB Hetzner server, resources are allocated as follows:

Kamal doesn't support Docker resource limits, so we rely on:

• Conservative JVM memory percentages
• Service-specific memory configuration
• Monitoring to detect resource contention

Kamal Integration & Production Deployment

Configuration for seamless deployment with Kamal, including monitoring and troubleshooting guidance.

Deployment Configuration

Persistent storage for logs and debugging data:

volumes:
  - "app_logs:/opt/app/logs"         # GC logs, heap dumps, application logs
  - "datomic_logs:/opt/datomic/logs" # Datomic transactor logs

Container and proxy-level health monitoring:

proxy:
  healthcheck:
    path: "/health"
    interval: 30s
    timeout: 10s

We rely on Kamal's proxy health checks rather than Docker HEALTHCHECK to avoid redundancy and curl dependencies in the container.

Adjusted for JVM warmup characteristics:

boot:
  limit: 1
  wait: 60  # Increased for JVM warmup and compilation

Production Monitoring

Access logs and debugging information:

# Monitor GC behavior
kamal gc-logs

# Check for heap dumps after OOM events
kamal heap-analysis

# View application logs
kamal logs

# Access Datomic transactor logs
kamal datomic-logs

Monitor these critical performance indicators:

• GC pause times (target: <200ms)
• Heap usage patterns
• Container memory utilization
• Database connection health

Troubleshooting Guide

OutOfMemoryError When the application runs out of memory:

1. Check heap dump in /opt/app/logs/
2. Analyze with tools like Eclipse MAT
3. Consider increasing MaxRAMPercentage if container has available memory

High GC Pause Times When garbage collection is impacting performance:

1. Review GC logs for pause time patterns
2. Consider adjusting MaxGCPauseMillis or G1HeapRegionSize
3. Monitor allocation rates in application code

Container Startup Issues When containers fail to start properly:

1. Check if health check endpoint is implemented
2. Verify volume mounts are accessible
3. Ensure secrets are properly configured

Performance Tuning Optimize based on production metrics:

• Monitor GC logs to identify allocation hotspots
• Use heap dumps to find memory leaks
• Adjust JVM parameters based on actual usage patterns
• Consider moving to JAR-based deployment for faster startup

Environment-Specific Settings

Development Environment:

• Disable direct linking for REPL development
• Use local Memcached instance
• Enable debug logging

Staging Environment:

• Mirror production JVM settings
• Use staging-specific service hostnames
• Enable additional monitoring

Production Environment:

• Full optimization enabled
• Persistent logging and heap dumps
• Resource-constrained settings for shared servers

Future Considerations

Potential improvements and scaling options for the Docker deployment strategy.

Scaling Options:

• Move to dedicated containers with hard resource limits
• Consider Kubernetes for more sophisticated resource management
• Implement horizontal scaling with load balancing

Performance Improvements:

• Evaluate JAR-based deployment for faster startup
• Consider GraalVM native image compilation
• Implement custom GC tuning based on production metrics