apps/studio/Dockerfile.dockerignore

@@ -0,0 +1,25 @@
+# Dockerfile-specific ignore for the STUDIO build ONLY (BuildKit picks up
+# `<dockerfile>.dockerignore` before the root .dockerignore).
+#
+# The root .dockerignore excludes `apps/studio/*` to keep the fork's ~660MB of
+# source out of the web/api build contexts. But the STUDIO image obviously
+# needs that source, so this file re-includes it while still stripping the
+# heavy/irrelevant stuff. web/api builds are unaffected (they fall back to the
+# root .dockerignore, which still excludes studio).
+node_modules
+**/node_modules
+.next
+**/.next
+.turbo
+**/.turbo
+dist
+**/dist
+.github
+.vscode
+.idea
+.DS_Store
+.env
+.env.*
+docs
+apps/runtime/data
+apps/api/data

docs/knowledge-base.md

@@ -785,3 +785,68 @@ Also written into project root **`CLAUDE.md`**. Skipping this step is a rule vio
 2. Doltgres = **Postgres family** (port 5432, `pg` driver). Missing feature ≠ abandon the product.
 3. Prefer workarounds (SQL rewrite, session settings, driver, version, config) over stock Postgres.
 4. **Notify flndrn** with: error text, which doc URL you used, options A/B/C — **before** switching architecture.
+
+## Doltgres version pinning + upgrade process (flndrn, 2026-08-01)
+
+**Rule:** Keep our Doltgres version in step with upstream. When DoltHub ships a new
+`dolthub/doltgresql` release, we evaluate and (after testing) move our pinned version
+along — because their releases carry the lock/concurrency/panic fixes we depend on.
+Do this as a **manual, tested process** — NOT an auto-updater/Watchtower/host-side poller
+(forbidden by `infra/CLAUDE.md`).
+
+- **Where the version is pinned:** `infra/dokploy/compose.dokploy.yml` — the `doltgres`,
+  `dolt-backup`, and init services all use `image: dolthub/doltgresql:<version>` (a readable
+  version tag, not an opaque `@sha256` digest, so the version is visible + bumpable).
+  Current pin: **`0.57.2`** (upgraded from `0.56.6` on 2026-08-01).
+- **Watch for new releases:** https://github.com/dolthub/doltgresql/releases and
+  https://github.com/dolthub/doltgresql/issues (customer-tagged issues show active fixes).
+- **Upgrade procedure (tested, no-guess):**
+  1. Read the new release notes for lock / GC / panic / connection-stability fixes.
+  2. **Validate data-read on a throwaway first:** `docker run` the new image on a fresh
+     volume (fresh-init), `cp -a` a copy of the live db dirs in, start, and confirm it reads
+     them (`select count(*) from information_schema.tables …` on `briven_control`). Doltgres
+     upgrades are forward-compatible in practice (0.56.6 → 0.57.2 read cleanly, no migration).
+  3. Set **`behavior.auto_gc_behavior.enable: false`** in `config.yaml` (see incident below).
+  4. Bump the tag in BOTH the repo compose and France `/code` compose, swap the prepared data
+     into the live volume, and recreate via `scripts/safe-redeploy-service.sh doltgres`.
+  5. Verify version (`select dolt_version()`), all DBs serve, API boots, then **watch it hold
+     under load** past the previous failure window before calling it done.
+
+## INCIDENT 2026-08-01 — Doltgres locked-under-load → whole platform 404 (ROOT-CAUSE FIXED)
+
+**Symptom:** `api.briven.tech` returns Traefik `404 page not found` on every route (incl.
+`/health`); `api` container crash-loops stuck at `applying migrations…`; Doltgres is "healthy"
+but serves **zero** databases — every `psql` connect (even `postgres`/`template1`) returns
+`FATAL: "database X does not exist"`, and trace logs show `database "briven_control" is locked
+by another dolt process … holds an exclusive write lock` **even with the server stopped and no
+other process alive**. Recurs within **4–50 minutes** of normal API write load. All 22 DB dirs
+on disk are intact (`.dolt/noms` valid) — **not** data loss.
+
+**Root cause:** Doltgres **`0.56.6`** (our pin) has a database-lock defect under concurrent
+write load. Upstream `0.57.1` shipped a **lock-subsystem rewrite** (`LockSubsystem.TryLock`)
+and `0.56.9` fixed an INSERT panic (a regression since `0.56.5` — our `0.56.6` was affected).
+`0.56.6` also predates the `auto_gc_behavior` config, and `0.57.1` **enabled auto-GC by default**
+(auto-GC "breaks all open connections"). NOT related to issue #2600 (that's app-level advisory
+locks). Verified against release notes, not guessed.
+
+**The SOLID fix (on Doltgres, per DOLTGRES-FIRST — no Postgres):**
+1. **Upgrade `0.56.6` → `0.57.2`** (validated: 0.57.2 reads 0.56.6 data unchanged; `briven_control`
+   51 tables, `briven_engine` 23 — no migration).
+2. **Disable auto-GC** in `config.yaml`: under `behavior:` add
+   `auto_gc_behavior:\n    enable: false` (only valid on 0.57.x — 0.56.6 rejects the key with
+   `field auto_gc_behavior not found`, which is how you know you must upgrade first).
+3. Recreate doltgres on 0.57.2 with the prepared data; restart API; watch it hold under load.
+
+**Emergency triage (temporary, if you must restore before upgrading):** a `cp -a` of each DB
+dir to **fresh inodes** clears the stale lock and the server serves again — but on `0.56.6` it
+re-jams under load within minutes-to-an-hour. It is NOT a fix; only the version upgrade is.
+Recipe that reliably serves: **fresh-init an EMPTY volume (entrypoint), then copy the DB dirs
+in** — a populated-but-uninitialized data-dir will NOT serve; and never leave `*.dirty`/rollback
+DB copies **inside** the data-dir (Doltgres enumerates every subdir and one bad copy aborts the
+whole multi-DB load).
+
+**Gotchas hit during the fix:** (a) the `dolt-backup` job's `dolt_backup(sync-url)` writes to
+every live DB and **accelerates the lock/GC failure** — keep it disabled until its method is
+reworked to cold snapshots; (b) a stray `docker run --rm` doltgres can linger and hold the
+volume lock — always `docker rm -f`; (c) load secrets into a shell var from `docker inspect …
+DOLTGRES_PASSWORD` / `.env.prod` — never literal in a command.

infra/backups/briven-backup.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=briven.tech — daily pg_dump of briven_control + briven_data to MinIO
+Description=briven.tech — daily DoltGres-native backup (all DBs, gentle/throttled) + pg_dump secondary + off-site mirror
 Documentation=https://code.konnos.org/flndrn/briven/src/branch/main/infra/backups
 Requires=docker.service
 After=docker.service network-online.target

infra/backups/briven-backup.sh

@@ -1,28 +1,38 @@
 #!/usr/bin/env bash
-# Daily backup of briven control plane (+ optional off-site mirror).
-# Runs on the France host via systemd (briven-backup.{service,timer}).
+# Daily backup of Briven — Doltgres-native (PRIMARY DR) + stock-Postgres pg_dump
+# (secondary rollback). Runs on the France host via systemd
+# (briven-backup.{service,timer}).
 #
-# Topology (2026-07-21): Briven is Doltgres end-to-end.
-#   - Control + data plane: DoltGres (briven_control + proj_* databases).
-#   - dolt-backup sidecar runs native dolt_backup for ALL databases including
-#     briven_control. This host script is a transitional dump helper for
-#     stock Postgres rollback windows only — prefer dolt_backup + off-site
-#     mirror of the doltgres_backups volume for real DR.
+# 2026-08-01 REWORK — why this changed:
+#   The old `dolt-backup` SIDECAR looped `dolt_backup('sync-url', …)` over EVERY
+#   database with no throttle, against a buggy DoltGres 0.56.6. That contributed
+#   to a full-platform outage (engine locked under load). Fix: engine upgraded to
+#   0.57.2 (lock-subsystem fixes) with auto-GC disabled, and backups are now a
+#   GENTLE, one-database-at-a-time, throttled pass run from this host job. The
+#   sidecar is removed from compose. Proven safe under monitoring (2026-08-01).
 #
-# Env (optional, /etc/briven/backup.env):
-#   BRIVEN_BACKUP_PG_CONTAINER   default: briven-brivenfrance-uilsk6-postgres-1
-#   BRIVEN_BACKUP_PG_USER        default: postgres
-#   BRIVEN_BACKUP_DBS            space-separated; default: briven_control
-#   BRIVEN_BACKUP_S3_ENDPOINT    if set with bucket+keys → off-site upload
-#   BRIVEN_BACKUP_S3_BUCKET
-#   BRIVEN_BACKUP_S3_ACCESS_KEY
-#   BRIVEN_BACKUP_S3_SECRET_KEY
+# Backup layers:
+#   1. PRIMARY  — `dolt_backup('sync-url', file:///backups/<db>)` for ALL live
+#                 DoltGres DBs (control + engine + every project) → doltgres_backups
+#                 volume (/backups). Restorable via `dolt backup restore`.
+#   2. auth.db  — snapshot the engine users/grants file (nothing else backs it up).
+#   3. SECONDARY— pg_dump of stock-Postgres briven_control (rollback window helper).
+#   4. OFF-SITE — mirror the doltgres_backups volume to external S3 (Backblaze/R2/…)
+#                 when BRIVEN_BACKUP_S3_* is configured (see BACKUP-OFFSITE.md).
 #
-# Local layout:
-#   /var/backups/briven/<db-name>/<YYYY-MM-DD>/<hh-mm-ss>.dump.gz
+# Env (/etc/briven/backup.env, optional):
+#   BRIVEN_DOLTGRES_CONTAINER          default: briven-brivenfrance-uilsk6-doltgres-1
+#   BRIVEN_DOLTGRES_BACKUPS_VOLUME     default: briven-brivenfrance-uilsk6_doltgres_backups
+#   BRIVEN_DOLTGRES_PASSWORD           default: read from the doltgres container env
+#   BRIVEN_BACKUP_THROTTLE_SECS        default: 8  (pause between DBs — gentleness)
+#   BRIVEN_BACKUP_PG_CONTAINER         default: briven-brivenfrance-uilsk6-postgres-1
+#   BRIVEN_BACKUP_PG_USER              default: postgres
+#   BRIVEN_BACKUP_PG_DBS               default: briven_control (secondary pg_dump)
+#   BRIVEN_BACKUP_S3_ENDPOINT/BUCKET/ACCESS_KEY/SECRET_KEY  → off-site mirror
+#   BRIVEN_BACKUP_LOCAL_RETENTION_DAYS default: 30
 #
-# Exit 1 if any off-site upload fails (local dump still kept) so
-# OnFailure=briven-backup-alert.service can fire.
+# Exit 1 if the dolt phase fails for any DB OR an off-site mirror fails, so
+# OnFailure=briven-backup-alert.service fires.
 
 set -euo pipefail
 
@@ -32,102 +42,135 @@ if [ -f "$BACKUP_ENV_FILE" ]; then
   source "$BACKUP_ENV_FILE"
 fi
 
+DC="${BRIVEN_DOLTGRES_CONTAINER:-briven-brivenfrance-uilsk6-doltgres-1}"
+BACKUPS_VOLUME="${BRIVEN_DOLTGRES_BACKUPS_VOLUME:-briven-brivenfrance-uilsk6_doltgres_backups}"
+THROTTLE="${BRIVEN_BACKUP_THROTTLE_SECS:-8}"
 PG_CONTAINER="${BRIVEN_BACKUP_PG_CONTAINER:-briven-brivenfrance-uilsk6-postgres-1}"
 PG_USER="${BRIVEN_BACKUP_PG_USER:-postgres}"
 # shellcheck disable=SC2206
-DBS=(${BRIVEN_BACKUP_DBS:-briven_control})
+PG_DBS=(${BRIVEN_BACKUP_PG_DBS:-briven_control})
 LOCAL_BACKUP_ROOT="/var/backups/briven"
 LOCAL_RETENTION_DAYS="${BRIVEN_BACKUP_LOCAL_RETENTION_DAYS:-30}"
 
 STAMP="$(date -u +'%Y-%m-%d/%H-%M-%S')"
-UPLOAD_FAILURES=0
-UPLOAD_FAILURE_DBS=""
-
-log() {
-  printf '[%s] %s\n' "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" "$*"
-}
-
-die() {
-  log "ERROR: $*"
-  exit 1
-}
-
-dump_one() {
-  local db_name="$1"
-  local local_dir="${LOCAL_BACKUP_ROOT}/${db_name}/${STAMP%/*}"
-  local local_file="${local_dir}/${STAMP##*/}.dump.gz"
-
-  mkdir -p "$local_dir"
-
-  if ! docker inspect "$PG_CONTAINER" >/dev/null 2>&1; then
-    die "postgres container not found: ${PG_CONTAINER}"
+STAMP_FLAT="$(date -u +'%Y-%m-%dT%H-%M-%SZ')"
+FAILURES=0
+FAILURE_DETAIL=""
+
+log()  { printf '[%s] %s\n' "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" "$*"; }
+die()  { log "ERROR: $*"; exit 1; }
+fail() { log "WARN: $*"; FAILURES=$((FAILURES + 1)); FAILURE_DETAIL="${FAILURE_DETAIL:+${FAILURE_DETAIL}; }$*"; }
+
+# --- resolve doltgres password without printing it ---
+dolt_password() {
+  if [ -n "${BRIVEN_DOLTGRES_PASSWORD:-}" ]; then
+    printf '%s' "$BRIVEN_DOLTGRES_PASSWORD"; return 0
   fi
+  docker inspect "$DC" --format '{{range .Config.Env}}{{println .}}{{end}}' 2>/dev/null \
+    | sed -n 's/^DOLTGRES_PASSWORD=//p' | head -1
+}
 
-  log "dumping ${db_name} from ${PG_CONTAINER}"
-  if ! docker exec "$PG_CONTAINER" pg_dump \
-        --username="$PG_USER" \
-        --format=custom \
-        --compress=0 \
-        --no-owner --no-privileges \
-        "$db_name" \
-        | gzip -9 > "$local_file"; then
-    rm -f "$local_file"
-    die "pg_dump failed for ${db_name}"
+# ===== PHASE 1: PRIMARY — gentle DoltGres-native backup of every live DB =====
+dolt_backup_all() {
+  docker inspect "$DC" >/dev/null 2>&1 || die "doltgres container not found: ${DC}"
+  local pw; pw="$(dolt_password)"
+  [ -n "$pw" ] || die "could not resolve DOLTGRES_PASSWORD"
+  local base="postgres://postgres:${pw}@127.0.0.1:5432"
+
+  log "dolt phase: enumerating live databases"
+  local dbs ok=0
+  dbs="$(docker exec "$DC" sh -lc \
+    "psql \"${base}/postgres?sslmode=disable\" -tAc \"select datname from pg_database where datname not in ('template0','template1','postgres')\"" \
+    2>/dev/null || true)"
+  [ -n "$dbs" ] || die "no databases enumerated (is doltgres serving?)"
+
+  for db in $dbs; do
+    # sync-url writes a restorable dolt archive to the server's /backups/<db>.
+    if docker exec "$DC" sh -lc \
+         "psql \"${base}/${db}?sslmode=disable\" -tAc \"select dolt_backup('sync-url','file:///backups/${db}')\"" \
+         >/dev/null 2>&1; then
+      log "  ok  dolt backup: ${db}"
+      ok=$((ok + 1))
+    else
+      fail "dolt backup failed: ${db}"
+    fi
+    sleep "$THROTTLE"   # gentleness — never hammer the engine
+  done
+  log "dolt phase: ${ok} database(s) backed up (throttle=${THROTTLE}s)"
+
+  # auth.db snapshot — engine users/grants; keep newest 14.
+  if docker exec "$DC" test -f /var/lib/doltgres/auth.db 2>/dev/null; then
+    if docker exec "$DC" sh -lc \
+         "mkdir -p /backups/auth-db && cp /var/lib/doltgres/auth.db /backups/auth-db/auth.db.${STAMP_FLAT} && ls -1t /backups/auth-db | tail -n +15 | while read -r f; do rm -f \"/backups/auth-db/\$f\"; done" \
+         >/dev/null 2>&1; then
+      log "  ok  auth.db snapshot"
+    else
+      fail "auth.db snapshot failed"
+    fi
   fi
+}
 
-  local size
-  size="$(stat -c%s "$local_file")"
-  log "ok ${db_name}: ${size} bytes → ${local_file}"
-
-  # Optional off-site (B2/R2 S3-compatible via minio/mc).
-  if [ -n "${BRIVEN_BACKUP_S3_ENDPOINT:-}" ] \
-     && [ -n "${BRIVEN_BACKUP_S3_BUCKET:-}" ] \
-     && [ -n "${BRIVEN_BACKUP_S3_ACCESS_KEY:-}" ] \
-     && [ -n "${BRIVEN_BACKUP_S3_SECRET_KEY:-}" ]; then
-    local object="s3://${BRIVEN_BACKUP_S3_BUCKET}/${db_name}/${STAMP}.dump.gz"
-    log "uploading ${object}"
-    local endpoint_no_scheme="${BRIVEN_BACKUP_S3_ENDPOINT#https://}"
-    endpoint_no_scheme="${endpoint_no_scheme#http://}"
-    if ! docker run --rm \
-          -v "${local_file}:/backup.dump.gz:ro" \
-          -e "MC_HOST_off=https://${BRIVEN_BACKUP_S3_ACCESS_KEY}:${BRIVEN_BACKUP_S3_SECRET_KEY}@${endpoint_no_scheme}" \
-          --entrypoint sh \
-          minio/mc:latest \
-          -c "mc cp /backup.dump.gz off/${BRIVEN_BACKUP_S3_BUCKET}/${db_name}/${STAMP}.dump.gz"; then
-      log "WARN: off-site upload failed for ${db_name} — local copy still safe"
-      UPLOAD_FAILURES=$((UPLOAD_FAILURES + 1))
-      UPLOAD_FAILURE_DBS="${UPLOAD_FAILURE_DBS:+${UPLOAD_FAILURE_DBS} }${db_name}"
+# ===== PHASE 2: SECONDARY — pg_dump of stock-Postgres (rollback window) =====
+pg_dump_secondary() {
+  docker inspect "$PG_CONTAINER" >/dev/null 2>&1 || { log "pg secondary: container ${PG_CONTAINER} absent — skipping"; return 0; }
+  for db in "${PG_DBS[@]}"; do
+    local dir="${LOCAL_BACKUP_ROOT}/${db}/${STAMP%/*}"
+    local file="${dir}/${STAMP##*/}.dump.gz"
+    mkdir -p "$dir"
+    if docker exec "$PG_CONTAINER" pg_dump --username="$PG_USER" --format=custom --compress=0 \
+         --no-owner --no-privileges "$db" 2>/dev/null | gzip -9 > "$file"; then
+      log "  ok  pg_dump ${db}: $(stat -c%s "$file") bytes"
     else
-      log "off-site upload ok"
+      rm -f "$file"; fail "pg_dump failed: ${db}"
     fi
-  else
-    log "off-site upload skipped (BRIVEN_BACKUP_S3_* unset) — Phase 0.1 still Not done"
-  fi
+  done
 }
 
-prune_local() {
-  log "pruning local dumps older than ${LOCAL_RETENTION_DAYS}d"
+# ===== PHASE 3: OFF-SITE — mirror the dolt backups volume to external S3 =====
+offsite_mirror() {
+  if [ -z "${BRIVEN_BACKUP_S3_ENDPOINT:-}" ] || [ -z "${BRIVEN_BACKUP_S3_BUCKET:-}" ] \
+     || [ -z "${BRIVEN_BACKUP_S3_ACCESS_KEY:-}" ] || [ -z "${BRIVEN_BACKUP_S3_SECRET_KEY:-}" ]; then
+    log "off-site mirror skipped (BRIVEN_BACKUP_S3_* unset) — see BACKUP-OFFSITE.md"
+    return 0
+  fi
+  local ep="${BRIVEN_BACKUP_S3_ENDPOINT#https://}"; ep="${ep#http://}"
+  local vol="/var/lib/docker/volumes/${BACKUPS_VOLUME}/_data"
+  [ -d "$vol" ] || { fail "off-site: backups volume path missing: ${vol}"; return 0; }
+  log "off-site mirror → s3://${BRIVEN_BACKUP_S3_BUCKET}/doltgres-backups/"
+  if docker run --rm -v "${vol}:/backups:ro" \
+        -e "MC_HOST_off=https://${BRIVEN_BACKUP_S3_ACCESS_KEY}:${BRIVEN_BACKUP_S3_SECRET_KEY}@${ep}" \
+        --entrypoint sh minio/mc:latest \
+        -c "mc mirror --overwrite --remove /backups off/${BRIVEN_BACKUP_S3_BUCKET}/doltgres-backups/" >/dev/null 2>&1; then
+    log "off-site mirror ok"
+  else
+    fail "off-site mirror failed"
+  fi
   if [ -d "$LOCAL_BACKUP_ROOT" ]; then
-    find "$LOCAL_BACKUP_ROOT" -type f -name '*.dump.gz' -mtime +"$LOCAL_RETENTION_DAYS" -delete || true
-    find "$LOCAL_BACKUP_ROOT" -type d -empty -delete || true
+    docker run --rm -v "${LOCAL_BACKUP_ROOT}:/pgd:ro" \
+      -e "MC_HOST_off=https://${BRIVEN_BACKUP_S3_ACCESS_KEY}:${BRIVEN_BACKUP_S3_SECRET_KEY}@${ep}" \
+      --entrypoint sh minio/mc:latest \
+      -c "mc mirror --overwrite /pgd off/${BRIVEN_BACKUP_S3_BUCKET}/pg-dumps/" >/dev/null 2>&1 \
+      || fail "off-site mirror (pg dumps) failed"
   fi
 }
 
-log "briven backup run starting (container=${PG_CONTAINER} dbs=${DBS[*]})"
-
-for db in "${DBS[@]}"; do
-  dump_one "$db"
-done
+prune_local() {
+  [ -d "$LOCAL_BACKUP_ROOT" ] || return 0
+  log "pruning local pg dumps older than ${LOCAL_RETENTION_DAYS}d"
+  find "$LOCAL_BACKUP_ROOT" -type f -name '*.dump.gz' -mtime +"$LOCAL_RETENTION_DAYS" -delete || true
+  find "$LOCAL_BACKUP_ROOT" -type d -empty -delete || true
+}
 
+log "briven backup run starting (doltgres=${DC}, throttle=${THROTTLE}s)"
+dolt_backup_all      # primary DR
+pg_dump_secondary    # secondary rollback
+offsite_mirror       # off-site (if configured)
 prune_local
 
-if [ "$UPLOAD_FAILURES" -gt 0 ]; then
-  log "ERROR: ${UPLOAD_FAILURES} off-site upload(s) failed (dbs=${UPLOAD_FAILURE_DBS})"
-  echo "upload_failures=${UPLOAD_FAILURES}" > /run/briven-backup-status
-  echo "upload_failure_dbs=${UPLOAD_FAILURE_DBS}" >> /run/briven-backup-status
+if [ "$FAILURES" -gt 0 ]; then
+  log "ERROR: ${FAILURES} failure(s): ${FAILURE_DETAIL}"
+  { echo "failures=${FAILURES}"; echo "detail=${FAILURE_DETAIL}"; } > /run/briven-backup-status
   exit 1
 fi
-
-# Clear stale status on success
 rm -f /run/briven-backup-status
-log "briven backup run complete"
+log "briven backup run complete — all layers ok"

infra/dokploy/compose.dokploy.yml

@@ -101,7 +101,7 @@ services:
   # into the named volume during the maintenance window). Never change this
   # path without checking `config.yaml` inside the volume.
   doltgres:
-    image: dolthub/doltgresql@sha256:0483137d0309598d3b0c111dff85d565077bd91cb0524ce00bb832929d5d5ddc
+    image: dolthub/doltgresql:0.57.2
     restart: unless-stopped
     logging: *briven-logging
     environment:
@@ -109,8 +109,10 @@ services:
       DOLTGRES_PASSWORD: ${BRIVEN_DOLTGRES_PASSWORD}
     volumes:
       - doltgres_data:/var/lib/doltgres
-      # Dolt-native backups land here (written by the server itself); the
-      # dolt-backup service triggers them. See that service for details.
+      # Dolt-native backups land here (written by the server itself); the host
+      # systemd job infra/backups/briven-backup.sh triggers them (gentle, one
+      # DB at a time). The old in-compose dolt-backup sidecar was REMOVED
+      # 2026-08-01 — its un-throttled per-DB loop helped lock DoltGres under load.
       - doltgres_backups:/backups
     healthcheck:
       # pg_isready ships in the doltgresql image and needs no password.
@@ -124,95 +126,18 @@ services:
     labels:
       - 'briven_logs=true'
 
-  # ─── data-plane backup ──────────────────────────────────────────────────
-  # REAL DoltGres backup (replaces the old placeholder sleep loop).
+  # ─── data-plane backup: moved to the host systemd job (2026-08-01) ──────
+  # The in-compose `dolt-backup` sidecar was REMOVED. Its un-throttled loop of
+  # `dolt_backup('sync-url', …)` over every DB helped lock DoltGres 0.56.6 under
+  # load (full-platform outage, see docs/knowledge-base.md → INCIDENT 2026-08-01).
+  # Backups now run from infra/backups/briven-backup.sh (systemd timer): a GENTLE,
+  # one-DB-at-a-time, throttled `dolt_backup` pass on the fixed 0.57.2 engine,
+  # plus off-site mirror. Restorable via `dolt backup restore`.
   #
-  # Why NOT pg_dump: tested 2026-06-26 against dolthub/doltgresql:latest
-  # (v0.56.6) — `pg_dump` aborts immediately with
-  #   "ERROR: SET TRANSACTION is not yet supported"
-  # because pg_dump opens a REPEATABLE READ READ ONLY snapshot transaction
-  # that DoltGres does not implement. So pg_dump CANNOT back up the data
-  # plane. (The control plane is real Postgres and is dumped separately by
-  # the host timers in infra/backups/.)
-  #
-  # What works (verified same day): Dolt's own backup, invoked over the
-  # Postgres wire with `SELECT dolt_backup('sync-url', '<file-url>')`. It
-  # writes a full, version-history-preserving Dolt archive (manifest +
-  # .darc) — re-running it re-syncs in place, so one backup dir per database
-  # already contains every commit (time-travel restore, not just a snapshot).
-  #
-  # This sidecar reuses the doltgresql image (it has `psql`), enumerates the
-  # data-plane databases each run, and asks the doltgres SERVER to back each
-  # one up into the shared `doltgres_backups` volume.
-  #
-  # OFF-SITE follow-up: mirroring the `doltgres_backups` volume to MinIO/B2/R2
-  # is done today by the host systemd timers in infra/backups/ (mc-based, see
-  # briven-backup.sh). Folding an `mc mirror /backups -> minio` step into this
-  # service needs an image carrying both psql and mc; tracked as a follow-up.
-  dolt-backup:
-    # Pinned to the SAME digest as the doltgres service (2026-07-07 window) —
-    # the sidecar's psql must always match the server's engine version.
-    image: dolthub/doltgresql@sha256:0483137d0309598d3b0c111dff85d565077bd91cb0524ce00bb832929d5d5ddc
-    restart: unless-stopped
-    logging: *briven-logging
-    depends_on:
-      doltgres:
-        condition: service_healthy
-    environment:
-      PGHOST: doltgres
-      PGPORT: '5432'
-      PGUSER: postgres
-      PGPASSWORD: ${BRIVEN_DOLTGRES_PASSWORD}
-      BRIVEN_BACKUP_INTERVAL_SECONDS: ${BRIVEN_BACKUP_INTERVAL_SECONDS:-86400}
-    volumes:
-      - doltgres_backups:/backups
-      # Read-only view of the server's data dir, ONLY so auth.db (the engine's
-      # users/grants file — corrupted once on 2026-07-07, nothing backed it up)
-      # can be snapshotted alongside the dolt backups below.
-      - doltgres_data:/doltgres-data:ro
-    entrypoint: ['/bin/sh', '-c']
-    command:
-      - |
-        set -eu
-        echo "dolt-backup: starting (interval=${BRIVEN_BACKUP_INTERVAL_SECONDS:-86400}s)"
-        while true; do
-          ts="$$(date -u +%Y-%m-%dT%H:%M:%SZ)"
-          echo "[dolt-backup $$ts] enumerating data-plane databases"
-          # All non-template databases on the doltgres server (one per project,
-          # plus the default `postgres`). -tA = tuples only, unaligned.
-          dbs="$$(psql -tA -d postgres -c \
-            "SELECT datname FROM pg_database WHERE datname NOT IN ('template0','template1')")"
-          for db in $$dbs; do
-            echo "[dolt-backup $$ts] backing up $$db -> file:///backups/$$db"
-            if psql -d "$$db" -c \
-                 "SELECT dolt_backup('sync-url', 'file:///backups/$$db');" >/dev/null; then
-              echo "[dolt-backup $$ts] ok $$db"
-            else
-              echo "[dolt-backup $$ts] WARN backup failed for $$db"
-            fi
-          done
-          # auth.db snapshot — tiny file, changes only on role/grant edits.
-          # Keep the newest 14 copies (2 weeks at the daily default interval).
-          if [ -f /doltgres-data/auth.db ]; then
-            mkdir -p /backups/auth-db
-            if cp /doltgres-data/auth.db "/backups/auth-db/auth.db.$$ts"; then
-              echo "[dolt-backup $$ts] ok auth.db snapshot"
-              ls -1t /backups/auth-db | tail -n +15 | while read -r old; do
-                rm -f "/backups/auth-db/$$old"
-              done
-            else
-              echo "[dolt-backup $$ts] WARN auth.db snapshot failed"
-            fi
-          else
-            echo "[dolt-backup $$ts] WARN auth.db not found in data dir"
-          fi
-          echo "[dolt-backup $$ts] run complete; sleeping"
-          sleep "$${BRIVEN_BACKUP_INTERVAL_SECONDS:-86400}"
-        done
-    networks:
-      - briven
-    labels:
-      - 'briven_logs=true'
+  # NOTE (still true): `pg_dump` CANNOT back up the DoltGres data plane — it opens
+  # a REPEATABLE READ READ ONLY snapshot txn DoltGres doesn't implement
+  # ("SET TRANSACTION is not yet supported"). Dolt-native `dolt_backup` is the
+  # only data-plane backup path; pg_dump is a stock-Postgres control-plane helper.
 
   # ─── briven-engine Auth DB (DOLTGRES ONLY — non-negotiable) ───────────
   # HARD RULE: the COMPLETE Briven project is Doltgres. New parts do not get
@@ -221,7 +146,7 @@ services:
   # briven-engine = Briven API code + tables in Doltgres DB `briven_engine`.
   # Create/migrate: API `ensureBrivenEngineDatabase` + schema bootstrap.
   briven-engine-db-init:
-    image: dolthub/doltgresql@sha256:0483137d0309598d3b0c111dff85d565077bd91cb0524ce00bb832929d5d5ddc
+    image: dolthub/doltgresql:0.57.2
     restart: 'no'
     logging: *briven-logging
     depends_on:

scripts/doltgres-version-check.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# Doltgres version tracker — compares our PINNED doltgresql image against the
+# latest upstream release, so we "move along" when DoltHub ships fixes.
+#
+# WHY: Doltgres is Beta; releases carry lock/concurrency/panic fixes we depend on
+# (see docs/knowledge-base.md → "INCIDENT 2026-08-01" — 0.56.6 locked under load,
+# fixed by upgrading to 0.57.2). This is a MANUAL, on-demand check — NOT a
+# Watchtower/host-side poller (forbidden by infra/CLAUDE.md). Run it locally or
+# from the brain cron; it never runs on the deploy host and never auto-upgrades.
+#
+# Usage:  bash scripts/doltgres-version-check.sh
+# Exit:   0 = up to date, 10 = newer release available, 1 = error.
+#
+# On a newer release: follow the tested upgrade procedure in
+# docs/knowledge-base.md → "Doltgres version pinning + upgrade process".
+
+set -euo pipefail
+
+COMPOSE="${BRIVEN_COMPOSE_FILE:-$(cd "$(dirname "$0")/.." && pwd)/infra/dokploy/compose.dokploy.yml}"
+
+# --- our pinned version (from the compose image tag) ---
+PINNED="$(grep -oE 'dolthub/doltgresql:[0-9]+\.[0-9]+\.[0-9]+' "$COMPOSE" | head -1 | cut -d: -f2 || true)"
+if [[ -z "$PINNED" ]]; then
+  echo "!! could not read a pinned dolthub/doltgresql:<version> tag from $COMPOSE"
+  echo "   (is it still pinned by @sha256 digest? switch to a version tag — see KB)"
+  exit 1
+fi
+
+# --- latest upstream release tag ---
+LATEST="$(curl -fsSL --max-time 20 https://api.github.com/repos/dolthub/doltgresql/releases/latest \
+  | grep -oE '"tag_name"[[:space:]]*:[[:space:]]*"v?[0-9]+\.[0-9]+\.[0-9]+"' \
+  | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1 || true)"
+if [[ -z "$LATEST" ]]; then
+  echo "!! could not fetch latest doltgresql release from GitHub"; exit 1
+fi
+
+echo "Doltgres pinned (ours): $PINNED"
+echo "Doltgres latest (upstream): $LATEST"
+
+# --- compare (sort -V) ---
+if [[ "$PINNED" == "$LATEST" ]]; then
+  echo "✓ up to date."
+  exit 0
+fi
+newest="$(printf '%s\n%s\n' "$PINNED" "$LATEST" | sort -V | tail -1)"
+if [[ "$newest" == "$PINNED" ]]; then
+  echo "✓ our pin is ahead of/equal to latest release (pre-release?). No action."
+  exit 0
+fi
+
+echo
+echo "⚠ NEWER Doltgres available: $PINNED -> $LATEST"
+echo "  Release notes: https://github.com/dolthub/doltgresql/releases/tag/v$LATEST"
+echo "  Open issues:   https://github.com/dolthub/doltgresql/issues"
+echo "  To upgrade: follow docs/knowledge-base.md → 'Doltgres version pinning + upgrade process'"
+echo "  (validate data-read on a throwaway first; keep auto_gc_behavior.enable:false)"
+exit 10