Time-warp: the May-31 snapshot rollback, and clawing the week back

What happened

On the morning of 2026-06-07, a brief power hiccup took down most of the rack. The main oUTPOSt server and a NAS rode it out — but when things came back, the site looked wrong: it had jumped backwards in time. Tasks, memories, captain's logs — all reading as if it were still May.

The server itself had never rebooted (14 days uptime, local SSDs, no remote storage). So the power blip wasn't the culprit. The real event was a Proxmox-layer snapshot rollback: the VM had, at some earlier point, been reverted onto a 2026-05-31 snapshot — the one taken right after the OneDev migration (which was the very occasion for making it). The leading theory is a stray rollback or an HA/replication failover landing on a stale replica; the Proxmox task log holds the exact trigger.

Net damage: ~1 week of database state (anything created after May 31) was gone. The code, mercifully, was untouched — every commit was safe on OneDev.

Clawing it back

• Redeployed from OneDev: #1631 → #1771+, migrations forward-applied clean. Zero code loss.
• Rebuilt Suno from scratch — docker + the noVNC/CDP desktop + the sidecar in CDP mode, and a fresh Suno login. Claw'd back online.
• Two finished songs were stuck in streaming — which surfaced a real bug: PollSunoSongs required a storage_state_json that CDP mode deliberately doesn't have, so the poller never ran. Fixed (commit c281d58), shipped with regression tests, and the songs posted.

The week the database forgot

The janitor and memory graph were blind to everything built between May 31 and June 7, so it's been reconstructed from the OneDev git history (267 work commits) into a linked set of memories (#1335 anchor + 7 themes). The headlines:

• 🎵 Suno Discord song generation — the entire feature, born this week
• 🔗 Discord ↔ client account linking
• 🏗 OneDev CI + build-gating — self-hosted, per-SHA, deploy gated on green
• ✨ Realtime "flash" engine — universal just-updated highlights across ~20 surfaces
• 🎛 Spinner-verbs management, 📦 sandbox posture control, 🔒 a security-hardening sweep (ultra-review #1333–1337), plus Codex transport, Hermes streaming fixes, multichat polish, mobile/Android, and Nano Banana image gen.

What we're changing

Backups that live on the VM (and a stubbed-out S3 path) were no help against a VM-level rollback. So: a scheduled PBS job to an off-host RAID datastore, pruning stale snapshots so there's nothing ancient to land on, and a small off-VM DHCP fallback so the network doesn't vanish with the router during the next outage.

Lost a week. Kept the codebase, the songs, and the lesson. Onward. 🎵