Tuesday, June 16, 2026
[Transparency Report #009][OPERATIONS] BGP Is Enabled! (Internally)
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What is happening?
This is a good thing for the exchange to have figured out. As of Jun 15th, we have learned
how to configure and enable BGP on OpnSense within the exchange. This means our techs
can now peer the exchange with member delegated /64s over /127 wireguard links! This is
a goal that we have been working towards, which also serves to get us moving towards our
goal of one day having a public ASN. Going forward, members who join the exchange will
have the option of having their /64 on-link or BGP peering with us an announcing their
/64 to our routing fabric.
Changes to the exchange:
- FurrIX Transit Fabric: Edge, Catos and Nardoragon are all peered using AS65300. Edge
announces a default route downstream, while the other two routers announce their assigned
/58s to the Edge.
- Exchange Member Peering: FurrIX has reserved AS65320 for peering with members of
our exchange, we also have started to rework our peering policies along with reserving
AS65400-65500 for member BGP sessions and AS65501-AS6550 for peering with other
hobbyist networks.
Changes Proposed:
Eventually FurrIX would like to add a BGP looking glass to our network that is peered
with the Edge that will should all ASNs and routes on the exchange, but this is a ways
off for the moment.
Are exchange operations affected?
Everything is operating normally, this was just quiet work in the background in order to
mature the exchange a little further and get to a point that we are reaching some of our
goals that were set for this year.
Thursday, June 11, 2026
[Transparency Report #008][OPERATIONS] OS Upgrades and House Keeping!
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What is happening?
FurrIX relies on a variety of open‑source operating systems and software projects that
work together to form the vIX. Periodically, our volunteers must perform housekeeping
tasks such as OS upgrades, NS zone updates and adjustments to routing and firewall
policies. Today marks the beginning of one such maintenance period focused on system
upgrades and lifecycle management.
What has been worked on so far:
- Core Router: Updated firewall rules to consolidate LIR‑range egress handling into a single
alias, reducing configuration duplication and improving maintainability
- Catos Access Router: Upgraded the OS due to end‑of‑life concerns and resolved an IPv6
routing issue affecting WireGuard interfaces
- Nardoragon Router: Upgraded the OS due to end‑of‑life concerns
Parts of the exchange still being worked on:
- Core Router: Pending OS upgrade to address EOL status
- Ikus vIX Access Router: Still offline and awaiting re‑deployment
- NMS: Monitoring is currently unavailable and requires full reconfiguration
- Status Page: Requires reconfiguration and an upgrade before it can be brought back online
Are exchange operations affected?
Yes — temporarily.
During house keeping, routing and service availability will be patchy as systems are updated
and rebooted. Once the work is complete, normal operations will resume as normal.
Monday, June 1, 2026
[Incident Report #035][DC] Power Failures
What Happened?
On June 1st at approximately 0145 EST, the FurrIX virtual exchange became unreachable. Shortly
after, our BGP announcements began withdrawing, causing our prefixes to disappear from upstream
looking glasses. Any members with devices tunneled into the exchange — phones, homelabs or
PCs — temporarily lost internet access and routing through the vIX.
We were seeing the following issues:
- vIX reachability — The virtual exchange was fully offline.
- NS1/NS2 Failure — Members could not reach either authoritative name server.
- Prefixes Left BGP — Our /48 and /44 announcements temporarily stopped.
- PHY One and Two power loss — Both hosts experienced unclean reboots.
- Backup failures — No backups were generated for June 1st.
What did we do to fix this?
We contacted the datacenter to determine the scope of the event and were informed
that WII were experiencing major power issues at the data center. Reviewing outage maps
for the region and weather reports, we became aware that severe thunderstorms passed
through the Kansas City area during the same time frame, which may have affected the
the region but we do not have concrete information on this right now.
As of this post, all FurrIX vIX services have recovered, our prefixes are visible in upstream
looking glasses again and member reachability has returned to normal.
Monday, May 25, 2026
[Transparency Report #007][OPERATIONS] Full Environment Rebuild Scheduled WIP
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What is happening?
The FurrIX vIX is currently going through its rebuild of our exchange and it is taking a little
longer than we expected. Due to a miscommunication, reinstalling the physical server’s OS
took a bit of time.
What has been reworked so far:
- Phy One: The ProxMox host has been rebuilt
- Core Router: We condensed our IPv6 edge and core router into one VM
- Nardoragon Router: Our services router is back online with new config
- Catos vIX Access Router: Has been pulled from backup and reconfigured
- NS1/Games-3P: These member facing services are back online
- Web Server: Our websites are back online
Parts of the exchange still being worked on:
- Mail-NG: the mail server has to be brought back online
- Ikus vIX Access Router: Secondary member facing router still being reconfig’d
- NMS: We currently have no monitoring, needs to be reconfigured
Are exchange operations affected?
Yes — temporarily.
During the rebuild window, routing and service availability will be null as systems are rebuilt
and renumbered. Once the work is complete, normal operations will resume with improved
stability, ease of expansion, better rooted upkeep and clarity.
Wednesday, May 20, 2026
[Incident Report #034][DNS] Inter‑subnet Communication Failure
What Happened?
On May 15th, our upstream data center completed a router upgrade. As an
unintended side effect, the FurrIX subnets located within the data center
were no longer able to reach one another. Because this issue was isolated to
internal data‑center paths, no external member traffic or internet‑facing name server
traffic were affected.
The issue went undetected until May 19th because our monitoring system and
our email system reside on opposite subnets. With inter‑subnet communication
broken, monitoring alerts could not reach us.
We were seeing the following issues:
- Internal service reachability — Some internal services were unreachable from
member connections. - NS2 isolation — Members could not reach NS2.
- Stale zones on NS2 — NS2 could not reach NS1; as a result, its zones
went stale on May 17th. - NMS visibility loss — The Network Management System could not reach devices
on PHY One for accounting and monitoring. - Backup failures — PHY One could not reach the PBS instance on PHY Two,
preventing nightly backups.
What did we do to fix this?
We provided the data center with test results and trace data confirming the inter‑subnet
routing failure. They corrected the configuration on their side, restoring full communication
between PHY One and PHY Two. All internal services, monitoring and backup operations
have returned to normal.