# Let's Encrypt Certificate Automation for Spring Boot REST Backends

**Reference implementation:** beta rest-web (`beta.web.tasking.rpp.cybercrucible.com`)
**Status:** Built, tested, and verified — including fail-closed security testing.

---

## 1. Overview

### What this replaces

Previously, backend Spring Boot servers used a long-lived self-signed certificate. nginx pinned that exact cert file via `proxy_ssl_trusted_certificate` and used `proxy_ssl_name "Unknown"` to match its CN. Every cert rotation required touching nginx on every front-end machine, so rotation effectively never happened.

### What this system does

Each backend machine issues and renews its own Let's Encrypt certificates using certbot (DNS-01 challenges via Route53), running entirely in Docker alongside the Spring app. Certificates hot-reload into the running JVM with no restart. nginx trusts the public CA root and verifies a shared "pool alias" name, so **nginx configuration never changes again** — adding backends, rotating certs, and promoting environments require zero nginx cert changes.

### How it runs (the closed loop)

```
certbot-renew container (12h loop)
  → no-op until cert has <30 days left
  → DNS-01 challenge via Route53 (TXT records, auto-cleaned)
  → new cert + fresh private key
  → deploy hook publishes flat PEM files (atomic mv)
  → shared Docker volume
  → Spring Boot file watcher swaps the cert live (no restart)
  → nginx: unaffected (trusts root, verifies alias)
```

Human involvement: zero, unless monitoring alerts.

### Key concepts

**The pool alias (e.g. `upstream.beta.web.tasking.rpp.cybercrucible.com`)** — an invented DNS name that exists only inside certificates and in nginx's `proxy_ssl_name`. It never gets an A/AAAA record and nothing ever connects to it. Every backend's cert contains two SANs: its own unique name, plus the shared alias. nginx verifies only the alias, which is why one static `proxy_ssl_name` works for every current and future backend in the pool. Required because `proxy_ssl_name` is one static value per location, while an upstream group has many servers.

**Alias naming convention:** `upstream.<domain-family>` — e.g. `upstream.beta.web.tasking...`, `upstream.web.tasking...` (prod), `upstream.agent.tasking...`. Keep aliases distinct per environment so a beta cert can never satisfy prod verification.

**DNS-01 challenges** — certbot proves domain control by creating a TXT record at `_acme-challenge.<domain>` via the Route53 API. Let's Encrypt never connects to any machine, so backends need no inbound exposure, no port 80, and IPv6-only containers work fine.

---

## 2. One-time company setup (already done — reference only)

### 2.1 Route53 hosted zone ID

Route 53 → Hosted zones → `cybercrucible.com` (public zone) → copy the Hosted Zone ID (`Z2RF8XABLU7FAI`).

### 2.2 IAM policy: `acme-dns01-cybercrucible`

IAM → Policies → Create policy → JSON. This is the **hardened** version (tested: certbot renewals succeed; A-record and non-challenge TXT writes are denied):

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListZones",
            "Effect": "Allow",
            "Action": "route53:ListHostedZones",
            "Resource": "*"
        },
        {
            "Sid": "GetChangeStatus",
            "Effect": "Allow",
            "Action": "route53:GetChange",
            "Resource": "arn:aws:route53:::change/*"
        },
        {
            "Sid": "AcmeTxtOnly",
            "Effect": "Allow",
            "Action": "route53:ChangeResourceRecordSets",
            "Resource": "arn:aws:route53:::hostedzone/Z2RF8XABLU7FAI",
            "Condition": {
                "ForAllValues:StringLike": {
                    "route53:ChangeResourceRecordSetsNormalizedRecordNames": [
                        "_acme-challenge.*"
                    ]
                },
                "ForAllValues:StringEquals": {
                    "route53:ChangeResourceRecordSetsRecordTypes": ["TXT"],
                    "route53:ChangeResourceRecordSetsActions": ["CREATE", "UPSERT", "DELETE"]
                }
            }
        }
    ]
}
```

The conditions restrict the key to TXT records named `_acme-challenge.*` only. A leaked key cannot touch A/AAAA/MX/CNAME records or move any traffic — worst case is an attacker can pass DNS-01 challenges (visible in public Certificate Transparency logs).

### 2.3 IAM user

IAM → Users → Create user (e.g. `acme-certbot`), **no console access**, attach the policy above, create an access key. Copy both values immediately (the secret shows once). This one credential serves all machines; create per-machine users later if individual revocation is wanted.

---

## 3. Spring Boot application requirements (once per app)

### 3.1 Version requirement

Spring Boot **3.2+** required for hot reload (`reload-on-update`). Verified working on Boot 4.1 with Netty/WebFlux; works identically on Tomcat.

### 3.2 One-time war/jar change

In the application's `application.properties`, **delete** all server-facing keystore lines:

```properties
# DELETE these — Boot refuses to combine them with SSL bundles:
server.ssl.key-store=...
server.ssl.key-store-password=...
server.ssl.key-store-type=...
```

Do **not** add bundle properties to the war — they are injected as environment variables from compose (Section 4.4), keeping TLS config deployment-side.

**Leave untouched** any keystore/truststore used for *client* connections (e.g. `rest-mongo-keystore.jks` for MongoDB TLS) — that is a separate concern from the server-facing HTTPS cert.

Rebuild the war.

---

## 4. Per-machine stack setup

One stack directory per machine. One certificate ("machine cert") covers **all** app containers on that machine — each container's domain plus the shared alias as SANs.

### 4.1 Directory layout

```
/var/<stackDirectory>/
├── compose.yml               # merged: app services + certbot services
├── certbot-deploy-hook.sh    # publishes flat PEMs (invariant)
├── .env                      # cert identity — the per-deployment file
├── aws.env                   # AWS credentials — chmod 600, NEVER in git
├── .gitignore                # contains: .env aws.env
├── rest-web.war              # the rebuilt application
├── <other app files>         # e.g. rest-mongo-keystore.jks
└── start.sh                  # guard + docker compose up -d
```

> **Lesson learned:** create `.env` and `aws.env` by editing directly (`nano`/`vi`), one `NAME=value` per line, no spaces around `=`. Do not paste heredoc blocks into editors — the commands end up inside the file and break compose parsing.

### 4.2 `.env`

```
CERT_NAME=rest-web-beta
PRIMARY_DOMAIN=river.beaver.beta.web.tasking.rpp.cybercrucible.com
ALIAS_DOMAIN=upstream.beta.web.tasking.rpp.cybercrucible.com
EXTRA_DOMAINS=
ACME_EMAIL=ops@cybercrucible.com
```

- `CERT_NAME` — certbot's local lineage name. Unique per stack, stable forever. Never appears in the cert itself.
- `PRIMARY_DOMAIN` — this machine's first container's own domain.
- `ALIAS_DOMAIN` — the shared pool alias. **Same value across every backend behind the same nginx domain family.**
- `EXTRA_DOMAINS` — additional `-d` flags for further containers on this machine, e.g. `EXTRA_DOMAINS=-d green.fox.beta.web.tasking.rpp.cybercrucible.com` (the `-d` is part of the value).

### 4.3 `aws.env` (chmod 600)

```
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
AWS_DEFAULT_REGION=us-east-1
AWS_USE_DUALSTACK_ENDPOINT=true
```

> **IPv6-only hosts (all our current backend machines):** the `AWS_USE_DUALSTACK_ENDPOINT=true` line is **mandatory** — Route53's default endpoint is IPv4-only and issuance fails with `Network unreachable` without it. Let's Encrypt's API is dual-stacked and needs nothing special.

### 4.4 `certbot-deploy-hook.sh` (chmod +x)

```bash
#!/bin/sh
set -eu
LIVE="/etc/letsencrypt/live/${CERT_NAME}"
DEST=/etc/letsencrypt/deployed
mkdir -p "$DEST"
cp -L "$LIVE/privkey.pem"   "$DEST/.privkey.tmp"   && chmod 600 "$DEST/.privkey.tmp"
cp -L "$LIVE/fullchain.pem" "$DEST/.fullchain.tmp" && chmod 644 "$DEST/.fullchain.tmp"
mv -f "$DEST/.privkey.tmp"   "$DEST/privkey.pem"
mv -f "$DEST/.fullchain.tmp" "$DEST/fullchain.pem"
echo "$(date -u +%FT%TZ) published ${CERT_NAME} to $DEST" >> /etc/letsencrypt/deploy.log
```

Why it exists: certbot's `live/` files are symlinks; on renewal only the symlink targets change, which the JVM's file watcher does not see. The hook publishes flat files; the temp-file + `mv` pattern is atomic, so the watcher never reads a half-written PEM.

### 4.5 `compose.yml`

The pattern: app services via a shared YAML anchor (so all instances inherit the cert plumbing), plus two certbot services.

```yaml
x-restServiceTemplate: &restServiceTemplate
    image: azul/zulu-openjdk:17
    working_dir: /var/<stackDirectory>/
    volumes:
      - ./rest-web.war:/var/<stackDirectory>/app.war:ro
      - ./rest-mongo-keystore.jks:/var/<stackDirectory>/rest-mongo-keystore.jks:ro
      - certs_data:/certs:ro
    environment:
      SERVER_SSL_BUNDLE: acme
      SPRING_SSL_BUNDLE_PEM_ACME_KEYSTORE_CERTIFICATE: "file:/certs/deployed/fullchain.pem"
      SPRING_SSL_BUNDLE_PEM_ACME_KEYSTORE_PRIVATE_KEY: "file:/certs/deployed/privkey.pem"
      SPRING_SSL_BUNDLE_PEM_ACME_RELOAD_ON_UPDATE: "true"
    command: ['java', '-Djava.net.preferIPv6Addresses=true', '-Djava.net.preferIPv4Stack=false', '-Djava.net.preferIPv6Stack=true', '-Xmx30g', '-jar', '/var/<stackDirectory>/app.war']
    restart: unless-stopped
    depends_on:
      certbot-init:
        condition: service_completed_successfully
    deploy:
      resources:
        limits:
          memory: 30G

services:
  <containerName1>:
    <<: *restServiceTemplate
    container_name: <containerName1>
    networks:
      public_macvlan_network:
        ipv6_address: "<container1 IPv6>"   # <container1 domain>

  certbot-init:
    image: certbot/dns-route53
    network_mode: host                 # REQUIRED on IPv6-only hosts
    env_file: ./aws.env
    environment:
      CERT_NAME: ${CERT_NAME}
    volumes:
      - certs_data:/etc/letsencrypt
      - ./certbot-deploy-hook.sh:/etc/letsencrypt/renewal-hooks/deploy/publish.sh:ro
    entrypoint: /bin/sh -c
    command: >
      "certbot certonly --dns-route53
      --cert-name ${CERT_NAME}
      -d ${PRIMARY_DOMAIN}
      -d ${ALIAS_DOMAIN} ${EXTRA_DOMAINS}
      --key-type ecdsa
      --non-interactive --agree-tos -m ${ACME_EMAIL}
      --keep-until-expiring
      && CERT_NAME=${CERT_NAME} /etc/letsencrypt/renewal-hooks/deploy/publish.sh"

  certbot-renew:
    image: certbot/dns-route53
    network_mode: host                 # REQUIRED on IPv6-only hosts
    env_file: ./aws.env
    environment:
      CERT_NAME: ${CERT_NAME}
    volumes:
      - certs_data:/etc/letsencrypt
      - ./certbot-deploy-hook.sh:/etc/letsencrypt/renewal-hooks/deploy/publish.sh:ro
    entrypoint: /bin/sh -c
    command: "'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done'"
    depends_on:
      certbot-init:
        condition: service_completed_successfully
    restart: unless-stopped

volumes:
  certs_data:

networks:
  public_macvlan_network:
    external: true
```

Design notes:

- **certbot-init** runs once per `compose up`: issues the cert if missing/expiring (`--keep-until-expiring` makes it a no-op otherwise), publishes the PEMs, exits. App containers wait for it via `depends_on: service_completed_successfully` — they either start with valid cert material guaranteed on disk, or don't start at all (fail-closed startup). It does NOT re-run on `docker compose restart <app>` — the PEMs persist in the volume, so restarts are fine.
- **certbot-renew** is the scheduler: checks every 12h, renews when <30 days remain, and certbot fires the deploy hook automatically on real renewals.
- **`network_mode: host` on both certbot services** — the compose default bridge is IPv4-only; on IPv6-only hosts, certbot gets `Errno 101 Network unreachable` without this. The certbot containers are outbound-only, so host networking adds no exposure.
- **Never add `--force-renewal` to the compose command** — LE's duplicate-cert limit is 5/week per identical SAN set; `--keep-until-expiring` keeps every `compose up` free.
- **Mount the tls directory, never individual files** — a single-file bind/volume mount pins the inode at container start and silently breaks hot reload after the first renewal.
- The AWS credentials go only to the certbot services, never the app containers.

### 4.6 `start.sh`

```bash
#!/bin/bash
[ -s "$(docker volume inspect <projectname>_certs_data -f '{{.Mountpoint}}' 2>/dev/null)/deployed/privkey.pem" ] 2>/dev/null || true
sudo docker compose up -d
```

(The `depends_on` gate makes an explicit PEM check optional — init guarantees the files before apps start. Keep start.sh as simply `sudo docker compose up -d` if preferred.)

---

## 5. First bring-up

### 5.1 DNS records

Create AAAA records in Route53 for each container's own domain → its macvlan IPv6. The alias domain gets **no** record, ever.

> **Lesson learned:** create the DNS record **before** any nginx configuration references the name (Section 6), and verify it resolves before reloading nginx.

### 5.2 Staging pass (recommended for new stacks)

Temporarily append `--staging` to the certbot-init command, then:

```bash
sudo ./start.sh
sudo docker compose logs -f certbot-init
```

Expected: account registered → cert requested for all SANs → "Successfully received certificate" → hook publish line → init exits 0 → app containers start.

Verify, then remove `--staging` and force one production reissue:

```bash
sudo docker compose run --rm certbot-init \
  "certbot certonly --dns-route53 --cert-name <CERT_NAME> \
   -d <PRIMARY_DOMAIN> -d <ALIAS_DOMAIN> <extra -d flags> \
   --key-type ecdsa --non-interactive --agree-tos -m <ACME_EMAIL> \
   --force-renewal && CERT_NAME=<CERT_NAME> /etc/letsencrypt/renewal-hooks/deploy/publish.sh"
```

> **Note on `docker compose run` with this compose file:** the service's entrypoint is already `/bin/sh -c`, so pass the whole command as ONE quoted string with no extra `sh -c` — double-wrapping drops you into an interactive shell instead of running the command.

### 5.3 Verify

```bash
# Cert content, from inside the container:
sudo docker exec <containerName> sh -c \
  "openssl x509 -in /certs/deployed/fullchain.pem -noout -issuer -ext subjectAltName"
# Want: issuer O=Let's Encrypt (no STAGING), all SANs listed.

# App started:
sudo docker logs <containerName> | grep -iE "started|error"
```

Troubleshooting decoder:

| Symptom | Cause |
|---|---|
| `Network unreachable` at LE step | Missing `network_mode: host` on certbot services |
| `Network unreachable` at Route53 step | Missing `AWS_USE_DUALSTACK_ENDPOINT=true` in aws.env |
| `AccessDenied` | IAM policy / zone ID mismatch |
| Zone not found | Domain spelling, or private zone instead of public |
| App boot error re: key-store + bundle | A `server.ssl.key-store*` line survived in the war (Section 3.2) |
| compose parse error on `.env` | Heredoc text pasted into the file (Section 4.1 note) |

---

## 6. nginx configuration (all front-end machines in the split pair)

In the domain's location blocks, replace the self-signed pin:

```nginx
# REMOVE:
#   proxy_ssl_trusted_certificate /etc/nginx/ssl/rest-web.crt;
#   proxy_ssl_name "Unknown";

# ADD:
proxy_ssl_verify              on;
proxy_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
proxy_ssl_verify_depth        3;
proxy_ssl_name                <ALIAS_DOMAIN>;
proxy_ssl_server_name         on;
proxy_ssl_session_reuse       on;
proxy_connect_timeout         5s;
```

- `ca-certificates.crt` is Ubuntu's built-in root store (includes ISRG Root X1, LE's root); it updates via `apt`. To pin trust to LE only: copy `/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt` and point at that instead (pin the **root**, never the intermediate — LE rotates intermediates).
- `verify_depth 3` covers leaf → LE intermediate → root.
- `proxy_connect_timeout 5s` — stopped macvlan containers **blackhole** (no RST); without this, failover to the surviving upstream waits the 60s default per attempt. 5s is ~1000× the LAN connect time. Keep long `proxy_read_timeout` values (e.g. SSE's 24h) — connect and read timeouts are independent.

Upstream block — domain names per existing convention, plus shared LB state:

```nginx
upstream <upstreamName> {
    zone <upstreamName> 64k;
    server "<container1 domain>:5050";
    server "<container2 domain>:5050";
}
```

- `zone` shares load-balancer state across worker processes. Without it, each worker keeps its own round-robin counter, and at low traffic (tests!) nearly all requests go to the first-listed server — this is normal nginx behavior, not a fault, but `zone` makes distribution sane at any traffic level and shares upstream health state across workers.
- Domain names in upstreams resolve **once, at config load**. Rule: the DNS record must exist and resolve *from inside the nginx container* before the reload, and any later IP change requires another reload.

Apply on **both** machines of the split pair, back-to-back:

```bash
sudo docker exec <nginx-container> nginx -t && sudo docker exec <nginx-container> nginx -s reload
# MANDATORY verification that the reload took:
sudo docker exec <nginx-container> nginx -T 2>/dev/null | grep -A8 "upstream <upstreamName>"
```

> **Lessons learned (nginx):**
> - **Always verify the loaded config with `nginx -T` after every reload.** A rejected reload (e.g. unresolvable upstream name) leaves nginx silently running the OLD config.
> - A stale config on one machine of the split pair = ~50% intermittent failures via DNS round-robin, which masquerades as flakiness.
> - Cutover timing: the moment a backend serves the LE cert, the old self-signed pin fails — change backend and both nginx machines in the same window.

---

## 7. Verification gauntlet (run for every new stack)

**All testing from a third machine (e.g. colo1) — NEVER from a macvlan host toward its own containers.** The kernel drops host↔own-macvlan traffic by design; testing from the host produces phantom hangs. (This caused significant confusion during the reference build.)

```bash
# 1. Cert on the wire, per backend:
echo | openssl s_client -6 -connect [<backend IPv6>]:5050 \
  -servername <ALIAS_DOMAIN> 2>/dev/null \
  | openssl x509 -noout -issuer -dates -ext subjectAltName
# Want: LE issuer, all SANs, ~90-day validity.
# NOTE: LE backdates notBefore by exactly 1 hour — a cert issued at 19:36 shows notBefore 18:36.

# 2. End-to-end through nginx:
curl https://<public domain>/public/health

# 3. Renewal machinery:
sudo docker compose exec certbot-renew certbot renew --dry-run

# 4. Rotation rehearsal (ONCE — consumes a duplicate-cert slot):
sudo docker compose exec certbot-renew certbot renew --force-renewal
sudo docker compose exec certbot-renew cat /etc/letsencrypt/deploy.log
# Then re-run check 1: notBefore ≈ now (minus 1h), container uptime UNCHANGED.
# Verify rotation ON THE WIRE — the Boot/Netty combo logs no greppable "reload" line.

# 5. Load-balancing check (volume matters — low-rate curls pin to one upstream without `zone`):
for i in $(seq 1 200); do curl -sS -m 8 https://<public domain>/public/health >/dev/null; done
# then count per-upstream in nginx access logs — expect a rough split.

# 6. Failover:
#    stop one app container; curls from the third machine should return Healthy via the
#    survivor (first affected request may pause ~proxy_connect_timeout). Restart it after.
```

### 7.1 Fail-closed security test (run once per environment)

Proves `proxy_ssl_verify` actually rejects forged certs:

```bash
# Swap in a self-signed impostor CLAIMING THE CORRECT SANs:
sudo docker compose exec certbot-renew sh -c '
  openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -days 1 \
    -keyout /etc/letsencrypt/deployed/privkey.pem.evil \
    -out    /etc/letsencrypt/deployed/fullchain.pem.evil \
    -subj "/CN=<PRIMARY_DOMAIN>" \
    -addext "subjectAltName=DNS:<PRIMARY_DOMAIN>,DNS:<ALIAS_DOMAIN>" ;
  cd /etc/letsencrypt/deployed ;
  cp privkey.pem privkey.pem.real ; cp fullchain.pem fullchain.pem.real ;
  mv privkey.pem.evil privkey.pem ; mv fullchain.pem.evil fullchain.pem'

# Wait ~30s (hot reload swaps in the impostor), then curl through nginx repeatedly:
# EXPECT 502 from BOTH front-end machines, and in nginx error logs:
#   "upstream SSL certificate verify error: (18:self-signed certificate)"

# Restore:
sudo docker compose exec certbot-renew sh -c '
  cd /etc/letsencrypt/deployed ;
  mv privkey.pem.real privkey.pem ; mv fullchain.pem.real fullchain.pem'
# Wait ~30s, curl → Healthy.
```

Reference build result: 502 on both machines, correct error logged, clean recovery — and the test doubles as two extra hot-reload demonstrations.

---

## 8. Monitoring (required, per stack)

Two alerts. **Test-fire both after wiring them** — an untested alert is a hope.

### 8.1 Served-cert expiry (<14 days) — runs on a third machine (colo1)

Checks what each backend **serves on the wire** (not certbot's files — the wire check catches every failure stage including hook-succeeded-but-app-didn't-reload). Certbot renews at 30 days out, so an alert means ~2 weeks of silent failures with ~2 weeks of runway.

`/opt/monitoring/check-cert-expiry.sh`, cron every 6h:

```bash
#!/bin/bash
THRESHOLD_SECONDS=$((14 * 24 * 3600))
alert() { curl -sS -X POST -H 'Content-Type: application/json' \
  -d "{\"text\":\"$1\"}" "<slack webhook URL>" >/dev/null; }

BACKENDS="
<backend1 IPv6>|5050|<label1>
<backend2 IPv6>|5050|<label2>
"
echo "$BACKENDS" | while IFS='|' read -r ip port label; do
  [ -z "$ip" ] && continue
  cert=$(echo | timeout 10 openssl s_client -6 -connect "[$ip]:$port" \
    -servername <ALIAS_DOMAIN> 2>/dev/null)
  if [ -z "$cert" ]; then
    alert ":rotating_light: CERT CHECK: cannot connect to $label at [$ip]:$port"
    continue
  fi
  if ! echo "$cert" | openssl x509 -noout -checkend "$THRESHOLD_SECONDS" >/dev/null 2>&1; then
    expiry=$(echo "$cert" | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
    alert ":rotating_light: CERT EXPIRING: $label expires $expiry (<14 days) — renewals failing"
  fi
done
```

Test-fire: set threshold to 90 days, run by hand, confirm the alert arrives, set back.

### 8.2 Renew-container alive — runs on each backend host

The renew container **is** the renewal scheduler; `restart: unless-stopped` covers crashes but not manual downs or daemon issues.

`/opt/monitoring/check-renew-container.sh`, cron every 6h:

```bash
#!/bin/bash
alert() { curl -sS -X POST -H 'Content-Type: application/json' \
  -d "{\"text\":\"$1\"}" "<slack webhook URL>" >/dev/null; }
CONTAINER="<stackdir>-certbot-renew-1"   # verify with: docker ps | grep renew
state=$(docker inspect -f '{{.State.Running}}' "$CONTAINER" 2>/dev/null)
if [ "$state" != "true" ]; then
  alert ":rotating_light: RENEW CONTAINER DOWN on $(hostname): $CONTAINER (state: ${state:-not found}) — renewals stopped"
fi
```

Test-fire: stop the container, run by hand, confirm alert, `docker compose up -d`.

---

## 9. Growth recipes

### 9.1 New app version

Replace the war → `sudo docker compose restart <containerName>`. Certs untouched; init does not re-run (PEMs persist in the volume).

### 9.2 Add another container on the same machine

1. Route53: AAAA record for the new domain → its new macvlan IPv6
2. `.env`: append to `EXTRA_DOMAINS` (e.g. `EXTRA_DOMAINS=-d green.fox.beta.web.tasking.rpp.cybercrucible.com`)
3. `compose.yml`: new service via `<<: *restServiceTemplate` + its own `container_name` and `ipv6_address` — it inherits cert mount, SSL env, and startup ordering from the anchor
4. **Explicit SAN-expansion reissue** (do not trust `compose up` alone — `--keep-until-expiring` may skip despite the changed domain list):

```bash
sudo docker compose run --rm certbot-init \
  "certbot certonly --dns-route53 --cert-name <CERT_NAME> \
   -d <PRIMARY_DOMAIN> -d <ALIAS_DOMAIN> -d <new domain> \
   --key-type ecdsa --non-interactive --agree-tos -m <ACME_EMAIL> \
   && CERT_NAME=<CERT_NAME> /etc/letsencrypt/renewal-hooks/deploy/publish.sh"
```

Certbot reissues into the same lineage and updates the renewal conf, so future auto-renewals carry all names. Running containers hot-reload the expanded cert. (Not a duplicate-cert rate-limit event — the SAN set changed.)
5. `sudo ./start.sh` (starts the new container)
6. nginx, both machines: one `server` line in the upstream block, reload, `nginx -T` verify. **No `proxy_ssl_*` changes.**
7. Verification gauntlet items 1, 2, 5.

### 9.3 New backend machine (availability pair)

1. Copy the stack directory pattern; adjust `working_dir`/volume paths
2. `.env`: **new** `CERT_NAME`, that machine's own domains, **same** `ALIAS_DOMAIN`
3. `aws.env`: same credential (dualstack line if IPv6-only host)
4. DNS records for its containers → `start.sh` (fresh lineage issues independently — no interaction with other machines' certs)
5. nginx upstream lines ×2, reload, verify
6. Full verification gauntlet

Machines' cert lifecycles are fully independent — one machine down does not affect the other's renewals. Wire the Section 8 monitoring for the new machine **the day it exists**.

### 9.4 New domain family (e.g. rest-agent, prod)

1. Pick its alias: `upstream.<family>.tasking.rpp.cybercrucible.com`
2. Per backend machine: this guide from Section 4, with all that machine's domains (`PRIMARY_DOMAIN` + `EXTRA_DOMAINS`) + the family alias
3. nginx: the Section 6 block with the new alias in `proxy_ssl_name`
4. Keep aliases distinct per family and per environment (beta vs prod)

---

## 10. Operational reference

### How renewal works

| Stage | What runs | On failure |
|---|---|---|
| Timer | certbot-renew loop, every 12h | Monitoring 8.2 |
| Check | no-op unless <30 days left | — |
| Renewal | DNS-01 via Route53, new cert + fresh key | Old cert still valid; ~60 retries over 30 days; monitoring 8.1 |
| Publish | deploy hook, atomic mv of flat PEMs | Old PEMs still served; monitoring 8.1 (wire check) |
| Reload | Spring watcher swaps SSLContext live | Old cert still served; monitoring 8.1; fix = container restart |

If a cert somehow fully expires: nginx verification fails closed (502, logged as "certificate has expired") until one successful renewal — recovery needs no restarts and no nginx changes. Reaching that state requires ~60 consecutive renewal failures plus ignoring the <14-day alert for two weeks plus ignoring LE's expiry emails.

### Command quick reference

```bash
# Lineage status:
sudo docker compose exec certbot-renew certbot certificates
# Deploy history:
sudo docker compose exec certbot-renew cat /etc/letsencrypt/deploy.log
# Cert being served (from a third machine):
echo | openssl s_client -6 -connect [<ip>]:5050 -servername <ALIAS_DOMAIN> 2>/dev/null \
  | openssl x509 -noout -issuer -dates -ext subjectAltName
# Loaded nginx config:
sudo docker exec <nginx-container> nginx -T 2>/dev/null | grep -A8 "upstream <name>"
```

### Rate limits (Let's Encrypt production)

- Duplicate certificates (identical SAN set): **5/week** — why `--force-renewal` is used at most once per stack for the rotation rehearsal, and never in compose
- Failed validations: 5/hostname/hour — a retry loop against a broken config can lock the shared alias for every backend; fix the cause, don't hammer
- Staging environment (`--staging`) has ~10× limits — use it for all experimentation

### Environment gotchas index (hard-won)

1. **IPv6-only hosts:** `network_mode: host` on certbot services + `AWS_USE_DUALSTACK_ENDPOINT=true`
2. **Test from a third machine only** — host↔own-macvlan is dropped by the kernel; testing from the host produces phantom hangs
3. **`nginx -T` verify after every reload** — rejected reloads silently keep the old config
4. **DNS record live and resolvable before nginx references the name** — upstream names resolve once, at load
5. **`zone` in upstream blocks** — without it, per-worker round-robin pins low-rate traffic to the first-listed server (normal behavior, confusing symptom)
6. **`proxy_connect_timeout 5s`** — stopped macvlan containers blackhole; default 60s makes failover appear broken
7. **LE backdates `notBefore` by 1 hour** — not a stale cert
8. **Verify rotation on the wire, not in app logs** — Boot/Netty logs no greppable reload line
9. **Mount the tls directory, never individual files** — single-file mounts pin the inode and silently kill hot reload
10. **Edit env files directly; never paste heredocs into them**
11. **`docker compose run` against these certbot services: pass the command as ONE quoted string** (entrypoint is already `sh -c`)