Monday, June 1, 2026
[Incident Report #035][DC] Power Failures
What are Incident Response Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. During the normal
operations of the exchange, our network and its supporting systems may encounter
operational defects, bugs, failed changes or attacks on our infrastructure. When this
happens, the FurrIX volunteers publish an incident response report to ensure all members
and peers remain informed as to what happened, how it went down and what we did to
recover or resolve the issue. As a hobbyist‑rooted vIX, we aim to keep communication clear,
accessible and practical to the best of our ability.
What Happened?
On June 1st at approximately 0145 EST, the FurrIX virtual exchange became unreachable. Shortly
after, our BGP announcements began withdrawing, causing our prefixes to disappear from upstream
looking glasses. Any members with devices tunneled into the exchange — phones, homelabs or
PCs — temporarily lost internet access and routing through the vIX.
We were seeing the following issues:
- vIX reachability — The virtual exchange was fully offline.
- NS1/NS2 Failure — Members could not reach either authoritative name server.
- Prefixes Left BGP — Our /48 and /44 announcements temporarily stopped.
- PHY One and Two power loss — Both hosts experienced unclean reboots.
- Backup failures — No backups were generated for June 1st.
What did we do to fix this?
We contacted the datacenter to determine the scope of the event and were informed
that WII were experiencing major power issues at the data center. Reviewing outage maps
for the region and weather reports, we became aware that severe thunderstorms passed
through the Kansas City area during the same time frame, which may have affected the
the region but we do not have concrete information on this right now.
As of this post, all FurrIX vIX services have recovered, our prefixes are visible in upstream
looking glasses again and member reachability has returned to normal.