Wednesday, July 15, 2026

[Incident Report #036][DC] Network Cabling Issues….

What Happened?
On July 14th, 2026 at around 2045EST, the FurrIX vIX went offline for about thirty-five
minutes. The volunteers at FurrIX reached out to the data center for some insight after
doing our own troubleshooting. It was discovered that our secondary server was still
alive but our primary had no response.

We were seeing the following issues:

  • vIX reachability — The virtual exchange was temporarily in a degraded state.
  • NS1 Failure — Members were relying on NS2’s zone cache temporarily.
  • MA BGP Failures — Our BGP sessions with members temporarily failed.
  • Web/email Failures - These services stopped responding.
  • Games-3P Failure - Game services went offline.

What did we do to fix this?
We contacted the data center to determine the scope of the event and were informed
that the techs at the upstream data center had found a loose network cable and that
was the cause of our server going offline. They have reseated the cable and the exchange
is now back online and services are reachable once again.

As of this post, all FurrIX vIX services have recovered.

Wednesday, July 1, 2026

[Transparency Report #012][Documentation] vIX Monitoring

What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.

What Happened
During the MFN to FurrIX migration, a number of larger infrastructure projects took priority
and were using our volunteer’s free time to get the exchange ready for full operation.
As a result, the monitoring stack (LibreNMS + graph export scripts) fell out of sync with the
new network layout. A stale firewall rule on PHY Two’s edge router blocked the monitoring
server’s requests with changes to new PI space, causing all transit graphs to stop updating.
Because this was a volunteer‑run transition with limited available time, the issue persisted
longer than usual, roughly four months, while other critical work was completed.

Changes to the exchange:
The outdated firewall rule was corrected, restoring connectivity between the web server
and LibreNMS. Once access was restored, all graph‑generation scripts came back online
and were patched with new tooling bits for extended monitoring internally and public
facing. All vIX flow‑rate graphs are now current and visible again.

Are exchange operations affected?
Both volunteers and members now have full visibility into how the vIX carries data and
how usage trends evolve over time. Aside from improved monitoring, normal operations
continue as expected.