Observability guide

This guide explains how logging and observability work in your ShipClojure application, where logs are stored, and how to query them for debugging and monitoring.

Overview

ShipClojure uses Telemere for structured logging. All application logs are written in JSON format, making them easy to search, filter, and analyze.

Key Features:

Structured JSON logging with request tracing
Automatic log rotation and compression
Easy querying with jq
Request ID tracking across all operations
30-day retention by default
Ready for log aggregation tools (Loki, ELK, CloudWatch, etc.)

Logging Architecture

Two Types of Logs

Your application produces two types of logs:

1. Application Logs (Structured JSON)

What: Your Clojure application code using Telemere Format: JSON (one object per line) Location:

Dev: logs/saas-app-dev.json (optional, disabled by default)
Prod: /var/log/shipclojure/saas-app.json

Example:

{
  "timestamp": "2025-10-16T06:30:22.287897Z",
  "level": "info",
  "id": "api-call/start",
  "msg": "POST /cqrs/command",
  "ns": "saas.logging",
  "line": 63,
  "data": {
    "request-method": "post",
    "uri": "/cqrs/command",
    "server-name": "localhost"
  },
  "ctx": {
    "req-id": "5efd277d-1ee7-46f8-94d7-93131c0d628a"
  },
  "error": null
}

2. Infrastructure Logs (Console/STDOUT)

What: Third-party libraries (Datomic, HTTP-Kit, Ring, etc.) Format: Routed through Telemere via SLF4J integration Location:

Dev: Console output (via Telemere's console handler)
Prod: Docker container logs (via kamal logs)

Log Storage

Development Environment

Console Logging:

Human-readable format printed to your terminal
Log levels configured in env/dev/clj/saas/env.clj
See Logging Guide for configuration details

Optional JSON Logging: To enable JSON logs in development, uncomment in env/dev/clj/saas/env.clj:

(logging/add-json-file-handler!
  {:path "logs/saas-app-dev.json"
   :interval :daily
   :max-file-size (* 1024 1024 50) ; 50MB
   :max-num-intervals 7})

Production Environment

JSON Application Logs:

Path: /var/log/shipclojure/saas-app.json
Rotation: Daily
Archives: /var/log/shipclojure/saas-app.json-2025-10-16d.1.gz
Retention: 30 days (configurable)
Max Size: 100MB per file before rotation

Infrastructure Logs:

Access via: kamal logs or kamal app logs --follow
Not persisted (unless Docker logging driver configured)

GC & JVM Logs:

Path: /var/log/shipclojure/gc-*.log
Heap dumps: /var/log/shipclojure/*.hprof (on OOM)

Accessing Logs

Quick Reference

# List log files
kamal json-logs-list

# View raw JSON logs (last 100 lines)
kamal json-logs-raw

# Live application logs (JSON, formatted)
kamal json-logs-tail

# All errors
kamal json-logs-errors

# Latest 100 logs
kamal json-logs-latest

# Today's logs
kamal json-logs-today

# Infrastructure logs (Datomic, HTTP-Kit, etc.)
kamal logs

# SSH into server
ssh root@YOUR_SERVER_IP

Kamal Log Aliases

Built-in aliases for common operations:

Alias	Description
`kamal json-logs-list`	List all files in logs directory
`kamal json-logs-raw`	Last 100 lines of logs (no formatting)
`kamal json-logs-tail`	Live tail of JSON logs with formatting
`kamal json-logs-errors`	Show all error-level logs
`kamal json-logs-latest`	Latest 100 logs, sorted by time
`kamal json-logs-today`	All logs from today
`kamal logs`	Infrastructure logs (STDOUT)
`kamal gc-logs`	JVM garbage collection logs
`kamal datomic-logs`	Datomic transactor logs
`kamal console`	SSH into container with bash

Direct Server Access

For advanced queries, SSH directly to the server and access the Docker volume:

ssh root@YOUR_SERVER_IP
cd /var/lib/docker/volumes/app_logs/_data

Note: Logs are stored in a Docker-managed volume, not a regular host directory. The volume is automatically created by Kamal and persists across container restarts.

Common Log Queries

All examples use jq for JSON parsing. The -s flag reads all lines into an array.

Note: Replace /var/lib/docker/volumes/app_logs/_data/saas-app.json with the actual path on your server, or use the Kamal aliases which handle the path automatically.

Show All Logs (Pretty-Printed)

cat /var/lib/docker/volumes/app_logs/_data/saas-app.json | jq -s '.'

Filter by Log Level

# All errors
cat /var/lib/docker/volumes/app_logs/_data/saas-app.json | jq -s '.[] | select(.level == "error")'

# All warnings and errors
cat /var/lib/docker/volumes/app_logs/_data/saas-app.json | jq -s '.[] | select(.level == "error" or .level == "warn")'

# Info and above
cat /var/lib/docker/volumes/app_logs/_data/saas-app.json | jq -s '.[] | select(.level != "debug")'

Filter by Time Range

# Today's logs
cat /var/lib/docker/volumes/app_logs/_data/saas-app.json | jq -s --arg date $(date +%Y-%m-%d) '.[] | select(.timestamp | startswith($date))'

# Logs from specific date
cat /var/lib/docker/volumes/app_logs/_data/saas-app.json | jq -s '.[] | select(.timestamp | startswith("2025-10-16"))'

# Last 100 entries, sorted by time
tail -100 /var/lib/docker/volumes/app_logs/_data/saas-app.json | jq -s 'sort_by(.timestamp)'

Filter by Event Type

# All API calls
cat /var/lib/docker/volumes/app_logs/_data/saas-app.json | jq -s '.[] | select(.id | startswith("api-call"))'

# Specific event
cat /var/lib/docker/volumes/app_logs/_data/saas-app.json | jq -s '.[] | select(.id == "api-call/response")'

# All nexus events
cat /var/lib/docker/volumes/app_logs/_data/saas-app.json | jq -s '.[] | select(.id | startswith("nexus"))'

Filter by User or Data

# Find logs for specific email
cat /var/log/shipclojure/saas-app.json | jq -s '.[] | select(.data.params."user/email" == "user@example.com")'

# Find signup attempts
cat /var/log/shipclojure/saas-app.json | jq -s '.[] | select(.data.params."command/kind" == ":command/sign-up")'

# Find POST requests to specific endpoint
cat /var/log/shipclojure/saas-app.json | jq -s '.[] | select(.data.uri == "/cqrs/command")'

Count Occurrences

# Count errors
cat /var/log/shipclojure/saas-app.json | jq -s '[.[] | select(.level == "error")] | length'

# Count by log level
cat /var/log/shipclojure/saas-app.json | jq -s 'group_by(.level) | map({level: .[0].level, count: length})'

# Count requests by endpoint
cat /var/log/shipclojure/saas-app.json | jq -s 'group_by(.data.uri) | map({uri: .[0].data.uri, count: length})'

Request Tracing

Every HTTP request gets a unique req-id that appears in all related logs. This allows you to trace a request's entire lifecycle.

Find All Logs for a Specific Request

# Replace with your actual request ID
export REQ_ID="5efd277d-1ee7-46f8-94d7-93131c0d628a"

cat /var/log/shipclojure/saas-app.json | jq -s --arg id "$REQ_ID" '.[] | select(.ctx."req-id" == $id)'

Trace a Request Timeline

cat /var/log/shipclojure/saas-app.json | jq -s --arg id "$REQ_ID" '
  [.[] | select(.ctx."req-id" == $id)]
  | sort_by(.timestamp)
  | .[]
  | {time: .timestamp, event: .id, msg: .msg}'

Example Request Trace

For a user signup request, you'll see:

api-call/start - Request received
api-call/params - Parameters logged (with sensitive data redacted)
nexus/logger - Database transaction
effects/send-email - Verification email sent
api-call/response - Response sent (with duration)

# Full trace with timing
cat /var/log/shipclojure/saas-app.json | jq -s --arg id "$REQ_ID" '
  [.[] | select(.ctx."req-id" == $id)]
  | sort_by(.timestamp)
  | .[]
  | {
      timestamp: .timestamp,
      event: .id,
      message: .msg,
      duration_ms: .data."saas.logging/ms"
    }'

Finding Request IDs

If you don't have a request ID, search for identifying information:

# Find request ID by email
cat /var/log/shipclojure/saas-app.json | jq -s '.[] | select(.data.params."user/email" == "user@example.com") | .ctx."req-id"' | head -1

Production Deployment

Log Locations on Server

Logs are stored in a Docker-managed volume at:

/var/lib/docker/volumes/app_logs/_data/
├── saas-app.json                    # Current application logs
├── saas-app.json-2025-10-16d.1.gz   # Daily archive (part 1)
├── saas-app.json-2025-10-16d.2.gz   # Daily archive (part 2)
├── saas-app.json-2025-10-15d.1.gz   # Previous day
├── gc-2025-10-16T06-30-00.log       # GC logs
└── java_pid12345.hprof              # Heap dump (if OOM occurred)

To find the exact location on your server:

docker volume inspect app_logs
# Look for "Mountpoint" in the output

Accessing Logs

Option 1: Kamal Aliases (Recommended)

kamal json-logs-list    # List all log files
kamal json-logs-errors  # Show errors
kamal json-logs-latest  # Latest 100 logs

Option 2: SSH + Direct Access

ssh root@YOUR_SERVER_IP
cd /var/lib/docker/volumes/app_logs/_data
cat saas-app.json | jq -s '.[] | select(.level == "error")'

Option 3: Download Logs Locally

# Download current logs
scp root@YOUR_SERVER_IP:/var/log/shipclojure/saas-app.json ./local-logs.json

# Query locally
cat local-logs.json | jq -s '.[] | select(.level == "error")'

Monitoring in Production

Check for errors regularly:

kamal json-logs-errors | jq -s '.[] | {time: .timestamp, msg: .msg, error: .error}'

Monitor response times:

cat /var/log/shipclojure/saas-app.json | jq -s '
  [.[] | select(.id == "api-call/response")]
  | map(.data."saas.logging/ms")
  | add / length'

Track failed requests (status >= 400):

cat /var/log/shipclojure/saas-app.json | jq -s '.[] | select(.data.status >= 400)'

Log Retention

Default Configuration

Rotation: Daily
Max File Size: 100MB (triggers rotation even mid-day)
Archives per Day: Up to 10 parts
Retention Period: 30 days
Compression: Gzip enabled

Archive Naming

saas-app.json-YYYY-MM-DDd.N.gz
                       │    └─ Part number (1-10)
                       └────── Daily interval marker

Customizing Retention

Edit env/prod/clj/saas/env.clj:

(logging/add-json-file-handler!
  {:path "logs/saas-app.json"
   :interval :daily           ; :daily, :weekly, or :monthly
   :max-file-size (* 1024 1024 100) ; 100MB
   :max-num-parts 10          ; Parts per interval
   :max-num-intervals 30})    ; Days/weeks/months to keep

Manual Cleanup

# On server, remove old archives
ssh root@YOUR_SERVER_IP
cd /var/log/shipclojure
rm saas-app.json-2025-09-*.gz

Custom Filters

Create a helper script for common queries:

scripts/query-logs.sh:

#!/bin/bash
LOG_FILE="/var/log/shipclojure/saas-app.json"

case "$1" in
  errors)
    cat $LOG_FILE | jq -s '.[] | select(.level == "error")'
    ;;
  slow-requests)
    cat $LOG_FILE | jq -s '.[] | select(.data."saas.logging/ms" > 1000)'
    ;;
  user)
    cat $LOG_FILE | jq -s --arg email "$2" '.[] | select(.data.params."user/email" == $email)'
    ;;
  *)
    echo "Usage: $0 {errors|slow-requests|user EMAIL}"
    ;;
esac

Performance Metrics from Logs

Average response time:

cat /var/log/shipclojure/saas-app.json | jq -s '
  [.[] | select(.id == "api-call/response") | .data."saas.logging/ms"]
  | add / length'

95th percentile response time:

cat /var/log/shipclojure/saas-app.json | jq -s '
  [.[] | select(.id == "api-call/response") | .data."saas.logging/ms"]
  | sort
  | .[length * 0.95 | floor]'

Request rate by endpoint:

cat /var/log/shipclojure/saas-app.json | jq -s '
  group_by(.data.uri)
  | map({uri: .[0].data.uri, count: length})
  | sort_by(.count)
  | reverse'

Troubleshooting

Logs Not Appearing

Check handler is registered:

;; In REPL
(require '[taoensso.telemere :as t])
(t/get-handlers)
;; Should show :json-file handler

Check file permissions:

ls -la /var/log/shipclojure/
# Should be owned by UID 1000 (clojure user)

Check disk space:

df -h /var/log/shipclojure

Invalid JSON Logs

JSON logs are written one object per line (newline-delimited JSON). Always use jq -s (slurp) to read all lines:

# Correct
cat logs.json | jq -s '.'

# Wrong - will fail on multi-line
cat logs.json | jq '.'

Finding Logs in Container

If you need to check logs inside the container:

kamal console
cd /opt/app/logs
ls -la
tail -f saas-app.json

Security Considerations

Sensitive Data Redaction

The following fields are automatically redacted from logs:

:password
:confirm-password
:token
:access-token
:refresh-token
:secret
:secret-key
:authorization
:csrf-token

Logged as:

{
  "data": {
    "params": {
      "user/password": "[REDACTED]"
    }
  }
}

Adding Custom Redaction

Edit src/clj/saas/logging.clj:

(def default-redact-key?
  #{:authorization :password :token :secret
    :api-key :credit-card :ssn}) ; Add more keys

Next Steps

Set up monitoring alerts based on error counts
Configure log aggregation for multi-server deployments
Create dashboards in Grafana/Kibana for visualizing metrics
Set up log-based alerting for critical errors

For more information on Telemere configuration, see the official documentation.

Getting Started

Tutorials

Backend

Frontend

Deployment

Observability guide

Overview

Logging Architecture

Two Types of Logs

1. Application Logs (Structured JSON)

2. Infrastructure Logs (Console/STDOUT)

Log Storage

Development Environment

Production Environment

Accessing Logs

Quick Reference

Kamal Log Aliases

Direct Server Access

Common Log Queries

Show All Logs (Pretty-Printed)

Filter by Log Level

Filter by Time Range

Filter by Event Type

Filter by User or Data

Count Occurrences

Request Tracing

Find All Logs for a Specific Request

Trace a Request Timeline

Example Request Trace

Finding Request IDs

Production Deployment

Log Locations on Server

Accessing Logs

Monitoring in Production

Log Retention

Default Configuration

Archive Naming

Customizing Retention

Manual Cleanup

Custom Filters

Performance Metrics from Logs

Troubleshooting

Logs Not Appearing

Invalid JSON Logs

Finding Logs in Container

Security Considerations

Sensitive Data Redaction

Adding Custom Redaction

Next Steps