Monday, June 1, 2026
[Incident Report #035][DC] Power Failures
What Happened?
On June 1st at approximately 0145 EST, the FurrIX virtual exchange became unreachable. Shortly
after, our BGP announcements began withdrawing, causing our prefixes to disappear from upstream
looking glasses. Any members with devices tunneled into the exchange — phones, homelabs or
PCs — temporarily lost internet access and routing through the vIX.
We were seeing the following issues:
- vIX reachability — The virtual exchange was fully offline.
- NS1/NS2 Failure — Members could not reach either authoritative name server.
- Prefixes Left BGP — Our /48 and /44 announcements temporarily stopped.
- PHY One and Two power loss — Both hosts experienced unclean reboots.
- Backup failures — No backups were generated for June 1st.
What did we do to fix this?
We contacted the datacenter to determine the scope of the event and were informed
that WII were experiencing major power issues at the data center. Reviewing outage maps
for the region and weather reports, we became aware that severe thunderstorms passed
through the Kansas City area during the same time frame, which may have affected the
the region but we do not have concrete information on this right now.
As of this post, all FurrIX vIX services have recovered, our prefixes are visible in upstream
looking glasses again and member reachability has returned to normal.
Monday, May 25, 2026
[Transparency Report #007][OPERATIONS] Full Environment Rebuild Scheduled WIP
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What is happening?
The FurrIX vIX is currently going through its rebuild of our exchange and it is taking a little
longer than we expected. Due to a miscommunication, reinstalling the physical server’s OS
took a bit of time.
What has been reworked so far:
- Phy One: The ProxMox host has been rebuilt
- Core Router: We condensed our IPv6 edge and core router into one VM
- Nardoragon Router: Our services router is back online with new config
- Catos vIX Access Router: Has been pulled from backup and reconfigured
- NS1/Games-3P: These member facing services are back online
- Web Server: Our websites are back online
Parts of the exchange still being worked on:
- Mail-NG: the mail server has to be brought back online
- Ikus vIX Access Router: Secondary member facing router still being reconfig’d
- NMS: We currently have no monitoring, needs to be reconfigured
Are exchange operations affected?
Yes — temporarily.
During the rebuild window, routing and service availability will be null as systems are rebuilt
and renumbered. Once the work is complete, normal operations will resume with improved
stability, ease of expansion, better rooted upkeep and clarity.
Wednesday, May 20, 2026
[Incident Report #034][DNS] Inter‑subnet Communication Failure
What Happened?
On May 15th, our upstream data center completed a router upgrade. As an
unintended side effect, the FurrIX subnets located within the data center
were no longer able to reach one another. Because this issue was isolated to
internal data‑center paths, no external member traffic or internet‑facing name server
traffic were affected.
The issue went undetected until May 19th because our monitoring system and
our email system reside on opposite subnets. With inter‑subnet communication
broken, monitoring alerts could not reach us.
We were seeing the following issues:
- Internal service reachability — Some internal services were unreachable from
member connections. - NS2 isolation — Members could not reach NS2.
- Stale zones on NS2 — NS2 could not reach NS1; as a result, its zones
went stale on May 17th. - NMS visibility loss — The Network Management System could not reach devices
on PHY One for accounting and monitoring. - Backup failures — PHY One could not reach the PBS instance on PHY Two,
preventing nightly backups.
What did we do to fix this?
We provided the data center with test results and trace data confirming the inter‑subnet
routing failure. They corrected the configuration on their side, restoring full communication
between PHY One and PHY Two. All internal services, monitoring and backup operations
have returned to normal.
Tuesday, May 19, 2026
[Transparency Report #006][OPERATIONS] Full Environment Rebuild Scheduled (May 22–23)
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What is happening?
FurrIX will be performing a full environment rebuild of its core infrastructure between
May 22nd and May 23rd. This maintenance window includes a comprehensive refresh
of the physical and virtual systems that support the exchange.
The following work is planned:
- Rebuilding the physical Proxmox host to ensure long‑term stability and alignment
with current operational standards
- Rebuilding all virtual routers, consolidating the routing layer down to three routers
for improved clarity and maintainability
- Transitioning the Games‑3P LXC container into a Games‑4P virtual machine, providing
better isolation and resource management
- Renumbering internal and service networks into FurrIX’s LIR‑assigned IPv6 ranges,
enabling the release of legacy datacenter‑provided address space
- General housekeeping tasks, including cleanup of unused resources, configuration
standardization and documentation updates
This work completes the post‑handoff cleanup following MFN’s network retirement and
removes legacy configurations that cannot be maintained in the current layout. It also
allow FurrIX the breathing room needed to ensure we can grow and control our operations
without legacy tooling getting in the way or having to rewrite large chunks of configuration.
Why this rebuild is needed
As FurrIX continues to mature into a vIX that tries to mimic the real world, several infrastructure
improvements are required to maintain operational clarity, reduce technical debt and ensure
long‑term sustainability:
- Infrastructure modernization: The current Proxmox host and routing layer have accumulated
legacy configurations from earlier phases of the project. A clean rebuild ensures consistency
and reliability.
- Routing simplification: Reducing to three routers improves manageability, reduces complexity
and aligns with the vIX’s current scale.
- Address space alignment: Moving fully into FurrIX’s LIR ranges allows the project to retire
datacenter‑assigned prefixes and operate with a clean, independent addressing plan.
- Service isolation: Migrating Games‑3P from an LXC container to a VM provides better performance
boundaries and operational flexibility.
- Operational hygiene: Housekeeping tasks ensure the environment remains maintainable and
well‑documented.
These changes support FurrIX’s goal of maintaining a clear governance boundary, a stable operational
footprint and a better run vIX while preserving its hobbyist openness.
What this means for members of the exchange
During the rebuild:
- Members may experience extended periods of routing instability or service unavailability during
the maintenance window. These interruptions will be minimized where possible, but the entire
vIX has to come offline for this work to be completed. We are aiming for no more than seven
hours of downtime, but have allotted two days in-case something unexpected happens.
- Member configuration changes will be required:
Member tunnel configurations, peering sessions and addressing assignments will require modification
and our volunteers will email new credentials and profiles soon as we are able to.
- No policy or governance changes:
This is strictly an infrastructure rebuild and does not affect membership, governance or peering policy.
Are exchange operations affected?
Yes — temporarily.
During the rebuild window, routing and service availability will be null as systems are rebuilt
and renumbered. Once the work is complete, normal operations will resume with improved
stability, ease of expansion, better rooted upkeep and clarity.
Thursday, May 14, 2026
[Transparency Report #005][OPERATIONS] Name servers moved into FurrIX
What are Transparency Reports?
As a community operated and governed virtual internet exchange, FurrIX has
to maintain and foster open and honest communication with our exchange
members. This means that from time to time, there will be items that come
up during our operations that could or do affect the exchange and our team
will publish notices in order to keep everyone in the know. As a community
internet exchange hobby project, FurrIX aims to have fully open communication
where practical.
What Happened?
FurrIX has completed the planned transition of its hybrid name servers from
MFN’s legacy network and DNS zone into full FurrIX operational control. This fulfills
the February 2026 agreement between MFN and FurrIX to migrate DNS operations while
maintaining continuity for all members. The move consolidates name server governance,
improves operational clarity and aligns DNS identity with the rest of the FurrIX
infrastructure.
We consulted with the operator of the newly scoped MFN project and mutually agreed to
leave their zone and glue records unchanged at the registry for now. Operationally, however,
FurrIX.zone no longer relies on NS entries from the MFN zone and the MFN operator is now
free to manage their DNS zone independently without requiring support from the FurrIX team.
Why this rework is needed:
With FurrIX taking over the network components previously operated under MFN, several
updates were required to maintain a clean governance boundary and preserve FurrIX’s
independent project status:
- Name server identity:
NS1 and NS2 now operate solely under the FurrIX namespace, retiring the hybrid MFN/FurrIX
state used during the transition period. (MFN’s NS entries remain available for their own use.)
- Operational ownership:
FurrIX now maintains full control of NS configuration, monitoring and security posture
completing the handoff from MFN and allowing the vIX to manage these servers as needed
- NS Records:
FurrIX NS glue records have been updated.
What this means for members of the exchange:
- No member impact —
This is a backend operational change and members should not experience any
difference in network behavior.
Are exchange operations affected?
Functionally, nothing member facing has changed.