Two weeks that began with a false alarm and ended with a courtroom. An "urgent" alert named a CDN-range IP as an attacker — the attribution was wrong, and untangling it produced a topology decision rather than a patch. That decision unlocked a unified blocklist subsystem and a cr...
Days after a snapshot rollback cost us a week, the station shipped two user-facing features back to back: a full Portal Songs experience — browse, play, create, and remix Suno songs from the web with no Discord needed — and a Song Studio AI Assist that drafts lyrics and styles fo...
A Proxmox snapshot rollback (not the power outage) warped the main server back to its May-31 migration snapshot, costing ~1 week of DB state. Code was safe on OneDev; we redeployed, rebuilt Suno in CDP mode, fixed the poller bug that stranded 2 songs, and reconstructed the lost w...