Thursday, August 20, 2026
Incident Report #039: Backup Server Upgrade Fail
What are Incident Response Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. During the normal
operations of the exchange, our network and its supporting systems may encounter
operational defects, bugs, failed changes or attacks on our infrastructure. When this
happens, the FurrIX volunteers publish an incident response report to ensure all members
and peers remain informed as to what happened, how it went down and what we did to
recover or resolve the issue. As a hobbyist‑rooted vIX, we aim to keep communication clear,
accessible and practical to the best of our ability.
What Happened
Our backup server had fallen significantly out of date across its software stack and during
an attempted upgrade. The update process broke roughly halfway through, leaving the system
in an unstable state and temporarily removing our ability to perform backups for the night.
Despite the failed upgrade, our volunteers successfully preserved all datastores before any data
loss could occur.
Upon assessing the severity of the failed upgrade, it became clear that attempting to salvage the
partially‑updated system would be too risky to FurrIX’s data. Our volunteers therefore used their
emergency governance override to authorize a full rebuild of the backup server. This decision allowed
us to move forward with a clean installation of the latest Proxmox Backup Server (PBS) version rather
than attempting to repair a heavily broken environment. Once the new PBS installation was in place,
the volunteers imported the preserved datastores and verified that the system could read all existing
backup data without any errors.
Points of Note
- Backups were unavailable for one night during the failure and rebuild window.
- No CT or VM backups appear to have been lost.
- All on‑hand data has been verified as readable on both PHYONE and PHYTWO.
- Normal backup operations have resumed under the rebuilt PBS environment.
The exchange avoided data loss and now has a fully updated and stable backup server. As always,
we appreciate the patience and understanding of our members and peers while we maintain and
improve the infrastructure that keeps FurrIX running.