Wednesday, July 15, 2026
[Incident Report #036][DC] Network Cabling Issues….
What Happened?
On July 14th, 2026 at around 2045EST, the FurrIX vIX went offline for about thirty-five
minutes. The volunteers at FurrIX reached out to the data center for some insight after
doing our own troubleshooting. It was discovered that our secondary server was still
alive but our primary had no response.
We were seeing the following issues:
- vIX reachability — The virtual exchange was temporarily in a degraded state.
- NS1 Failure — Members were relying on NS2’s zone cache temporarily.
- MA BGP Failures — Our BGP sessions with members temporarily failed.
- Web/email Failures - These services stopped responding.
- Games-3P Failure - Game services went offline.
What did we do to fix this?
We contacted the data center to determine the scope of the event and were informed
that the techs at the upstream data center had found a loose network cable and that
was the cause of our server going offline. They have reseated the cable and the exchange
is now back online and services are reachable once again.
As of this post, all FurrIX vIX services have recovered.
Wednesday, July 1, 2026
[Transparency Report #012][Documentation] vIX Monitoring
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What Happened
During the MFN to FurrIX migration, a number of larger infrastructure projects took priority
and were using our volunteer’s free time to get the exchange ready for full operation.
As a result, the monitoring stack (LibreNMS + graph export scripts) fell out of sync with the
new network layout. A stale firewall rule on PHY Two’s edge router blocked the monitoring
server’s requests with changes to new PI space, causing all transit graphs to stop updating.
Because this was a volunteer‑run transition with limited available time, the issue persisted
longer than usual, roughly four months, while other critical work was completed.
Changes to the exchange:
The outdated firewall rule was corrected, restoring connectivity between the web server
and LibreNMS. Once access was restored, all graph‑generation scripts came back online
and were patched with new tooling bits for extended monitoring internally and public
facing. All vIX flow‑rate graphs are now current and visible again.
Are exchange operations affected?
Both volunteers and members now have full visibility into how the vIX carries data and
how usage trends evolve over time. Aside from improved monitoring, normal operations
continue as expected.
Thursday, June 25, 2026
[Transparency Report #011][Documentation] Bringing our Status Pages Back
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What is happening?
Our volunteers have been working the past few days on getting our status pages back
online so that our members can keep an eye on the exchange and get a heads up if
a service, router or endpoint goes down. This is being done as part of our transparency
towards the members that make up and run our exchange with us. While you won’t get
hard numbers from the Status Bot, it will give you a basic idea of the exchange’s health.
Changes to the exchange:
- status.furrix.zone has been brought back online
- addresses and checks have been updated to the new /44
- alerting rules will forward problems to our internal discord
Are exchange operations affected?
Nothing inside the exchange has been modified, this work is just bolt on monitoring for
peace of mind and is an augment to our upcoming LibreNMS deployment.
Saturday, June 20, 2026
[Transparency Report #010][OPERATIONS] BGP gets a Tune Up!
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What is happening?
This week’s changes focused on tightening routing policy between Edge, our
member access routers and the services router (Catos, Ikus and Nardoragon).
The goal was simple:
- eliminate any possibility of route leaks
- enforce strict prefix‑origination rules
- ensure the exchange remains hobbyist‑grade, stable, and predictable
All required changes were applied without service interruption. All member routes
remained visible and stable throughout the transition.
Changes to the exchange:
Our volunteers have implemented uniform BGP filtering across all internal routers.
Catos, Ikus, and Nardoragon:
- May only advertise their assigned /58 prefixes
- May only learn the default route from Edge
- Cannot advertise our PI /45 or /46 aggregate anywhere
- Cannot learn leaked routes from Edge or from each other
Edge:
- Only advertises ::/0 toward all downstream routers
- Only accepts each downstream router’s assigned /58
- Is the only router permitted to originate the /45 and /46 aggregates
- Will only originate those aggregates once we obtain our own ASN (maps already in place)
Prefix‑lists and route‑maps have been standardized across all routers to ensure the fabric
remains predictable and safe for our volunteers and members to continue learning and
experimenting within the exchange. This includes consistent permit/deny ordering, strict
prefix matching and hardened default‑deny behavior.
Are exchange operations affected?
Everything is operating as expected. This was much‑needed work in the background to ensure
long‑term stability and predictability of the exchange. These changes make our BGP setup more
oops‑proof, better hardened and more aligned with real IX operational practices — while still
keeping the environment friendly for hobbyist experimentation.
Tuesday, June 16, 2026
[Transparency Report #009][OPERATIONS] BGP Is Enabled! (Internally)
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What is happening?
This is a good thing for the exchange to have figured out. As of Jun 15th, we have learned
how to configure and enable BGP on OpnSense within the exchange. This means our techs
can now peer the exchange with member delegated /64s over /127 wireguard links! This is
a goal that we have been working towards, which also serves to get us moving towards our
goal of one day having a public ASN. Going forward, members who join the exchange will
have the option of having their /64 on-link or BGP peering with us an announcing their
/64 to our routing fabric.
Changes to the exchange:
- FurrIX Transit Fabric: Edge, Catos and Nardoragon are all peered using AS65300. Edge
announces a default route downstream, while the other two routers announce their assigned
/58s to the Edge.
- Exchange Member Peering: FurrIX has reserved AS65320 for peering with members of
our exchange, we also have started to rework our peering policies along with reserving
AS65400-65500 for member BGP sessions and AS65501-AS6550 for peering with other
hobbyist networks.
Changes Proposed:
Eventually FurrIX would like to add a BGP looking glass to our network that is peered
with the Edge that will should all ASNs and routes on the exchange, but this is a ways
off for the moment.
Are exchange operations affected?
Everything is operating normally, this was just quiet work in the background in order to
mature the exchange a little further and get to a point that we are reaching some of our
goals that were set for this year.