Sunday, August 2, 2026
[Incident Report #038][vIX] NS Abuse Impacting Exchange Stability
What are Incident Response Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. During the normal
operations of the exchange, our network and its supporting systems may encounter
operational defects, bugs, failed changes or attacks on our infrastructure. When this
happens, the FurrIX volunteers publish an incident response report to ensure all members
and peers remain informed as to what happened, how it went down and what we did to
recover or resolve the issue. As a hobbyist‑rooted vIX, we aim to keep communication clear,
accessible and practical to the best of our ability.
What Happened?
During August 1st and August 2nd, 2026, the FurrIX vIX experienced a significant
service disruption caused by an unexpected surge of traffic hammering our name
servers. This traffic overwhelmed our exchange fabric, resulting in instability and
degraded performance for multiple services- up to the point of making the exchange
itself unreachable and completely saturated our link to our upstream.
During the events, we have been observing:
- vIX reachability issues — The virtual exchange entered a degraded state
due to excessive inbound NS load.
- NS1/NS2 saturation — Both name servers experienced heavy recursive query
pressure, exhausting logging capacity and reducing responsiveness.
- MA BGP instability — Member BGP sessions dropped from the vIX due to excessive traffic.
- Monitoring failures — NMS monitoring temporarily dropped during peak saturation.
- General service degradation — Web, email and auxiliary services were intermittently unreachable.
This event originated from external recursive DNS traffic and did not involve any FurrIX members.
Upon reviewing logs, traffic alerts and our periodic graphs, it was determined this may be automated
scanner traffic that found and exploited our infrastructure. Our volunteers are currently making
updated procedures and are in the process of updating the vIX configuration to deal with this.
What are we doing to fix this?
During investigation, it was determined that our existing rate‑limit configuration inside Proxmox
did not behave as we originally thought they would and thus did not protect the exchange fabric
from inbound saturation. Furthermore, the traffic that slapped the exchange was rotating through
IPv6 addresses at a rate and pattern that mostly avoided our BIND9 rate limits, which has been
eye opening in itself.
To address this, FurrIX is implementing the following corrective actions:
- Migrating all rate‑limiting from Proxmox to the OpnSense router VMs, where
shaping and policing can be applied correctly at the network demarcation.
- Reworking our NS policy, including tightening recursion rules, implementing
stricter query controls and improving protections for authoritative services.
- Enhancing network monitoring, adding visibility at the router, VM and hypervisor
layers to detect similar events earlier; including active alerting to our Discord.
Exchange operations may remain unstable while volunteers complete these changes and validate
the new configuration.
Current Status
As of this post, the exchange fabric has recovered and all core FurrIX vIX services are reachable.
Additional hardening work is ongoing and further updates will be posted once our volunteers have
completed all the ongoing work.
Saturday, June 20, 2026
[Transparency Report #010][OPERATIONS] BGP gets a Tune Up!
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What is happening?
This week’s changes focused on tightening routing policy between Edge, our
member access routers and the services router (Catos, Ikus and Nardoragon).
The goal was simple:
- eliminate any possibility of route leaks
- enforce strict prefix‑origination rules
- ensure the exchange remains hobbyist‑grade, stable, and predictable
All required changes were applied without service interruption. All member routes
remained visible and stable throughout the transition.
Changes to the exchange:
Our volunteers have implemented uniform BGP filtering across all internal routers.
Catos, Ikus, and Nardoragon:
- May only advertise their assigned /58 prefixes
- May only learn the default route from Edge
- Cannot advertise our PI /45 or /46 aggregate anywhere
- Cannot learn leaked routes from Edge or from each other
Edge:
- Only advertises ::/0 toward all downstream routers
- Only accepts each downstream router’s assigned /58
- Is the only router permitted to originate the /45 and /46 aggregates
- Will only originate those aggregates once we obtain our own ASN (maps already in place)
Prefix‑lists and route‑maps have been standardized across all routers to ensure the fabric
remains predictable and safe for our volunteers and members to continue learning and
experimenting within the exchange. This includes consistent permit/deny ordering, strict
prefix matching and hardened default‑deny behavior.
Are exchange operations affected?
Everything is operating as expected. This was much‑needed work in the background to ensure
long‑term stability and predictability of the exchange. These changes make our BGP setup more
oops‑proof, better hardened and more aligned with real IX operational practices — while still
keeping the environment friendly for hobbyist experimentation.
Tuesday, June 16, 2026
[Transparency Report #009][OPERATIONS] BGP Is Enabled! (Internally)
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What is happening?
This is a good thing for the exchange to have figured out. As of Jun 15th, we have learned
how to configure and enable BGP on OpnSense within the exchange. This means our techs
can now peer the exchange with member delegated /64s over /127 wireguard links! This is
a goal that we have been working towards, which also serves to get us moving towards our
goal of one day having a public ASN. Going forward, members who join the exchange will
have the option of having their /64 on-link or BGP peering with us an announcing their
/64 to our routing fabric.
Changes to the exchange:
- FurrIX Transit Fabric: Edge, Catos and Nardoragon are all peered using AS65300. Edge
announces a default route downstream, while the other two routers announce their assigned
/58s to the Edge.
- Exchange Member Peering: FurrIX has reserved AS65320 for peering with members of
our exchange, we also have started to rework our peering policies along with reserving
AS65400-65500 for member BGP sessions and AS65501-AS6550 for peering with other
hobbyist networks.
Changes Proposed:
Eventually FurrIX would like to add a BGP looking glass to our network that is peered
with the Edge that will should all ASNs and routes on the exchange, but this is a ways
off for the moment.
Are exchange operations affected?
Everything is operating normally, this was just quiet work in the background in order to
mature the exchange a little further and get to a point that we are reaching some of our
goals that were set for this year.
Thursday, June 11, 2026
[Transparency Report #008][OPERATIONS] OS Upgrades and House Keeping!
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What is happening?
FurrIX relies on a variety of open‑source operating systems and software projects that
work together to form the vIX. Periodically, our volunteers must perform housekeeping
tasks such as OS upgrades, NS zone updates and adjustments to routing and firewall
policies. Today marks the beginning of one such maintenance period focused on system
upgrades and lifecycle management.
What has been worked on so far:
- Core Router: Updated firewall rules to consolidate LIR‑range egress handling into a single
alias, reducing configuration duplication and improving maintainability
- Catos Access Router: Upgraded the OS due to end‑of‑life concerns and resolved an IPv6
routing issue affecting WireGuard interfaces
- Nardoragon Router: Upgraded the OS due to end‑of‑life concerns
Parts of the exchange still being worked on:
- Core Router: Pending OS upgrade to address EOL status
- Ikus vIX Access Router: Still offline and awaiting re‑deployment
- NMS: Monitoring is currently unavailable and requires full reconfiguration
- Status Page: Requires reconfiguration and an upgrade before it can be brought back online
Are exchange operations affected?
Yes — temporarily.
During house keeping, routing and service availability will be patchy as systems are updated
and rebooted. Once the work is complete, normal operations will resume as normal.
Tuesday, May 19, 2026
[Transparency Report #006][OPERATIONS] Full Environment Rebuild Scheduled (May 22–23)
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What is happening?
FurrIX will be performing a full environment rebuild of its core infrastructure between
May 22nd and May 23rd. This maintenance window includes a comprehensive refresh
of the physical and virtual systems that support the exchange.
The following work is planned:
- Rebuilding the physical Proxmox host to ensure long‑term stability and alignment
with current operational standards
- Rebuilding all virtual routers, consolidating the routing layer down to three routers
for improved clarity and maintainability
- Transitioning the Games‑3P LXC container into a Games‑4P virtual machine, providing
better isolation and resource management
- Renumbering internal and service networks into FurrIX’s LIR‑assigned IPv6 ranges,
enabling the release of legacy datacenter‑provided address space
- General housekeeping tasks, including cleanup of unused resources, configuration
standardization and documentation updates
This work completes the post‑handoff cleanup following MFN’s network retirement and
removes legacy configurations that cannot be maintained in the current layout. It also
allow FurrIX the breathing room needed to ensure we can grow and control our operations
without legacy tooling getting in the way or having to rewrite large chunks of configuration.
Why this rebuild is needed
As FurrIX continues to mature into a vIX that tries to mimic the real world, several infrastructure
improvements are required to maintain operational clarity, reduce technical debt and ensure
long‑term sustainability:
- Infrastructure modernization: The current Proxmox host and routing layer have accumulated
legacy configurations from earlier phases of the project. A clean rebuild ensures consistency
and reliability.
- Routing simplification: Reducing to three routers improves manageability, reduces complexity
and aligns with the vIX’s current scale.
- Address space alignment: Moving fully into FurrIX’s LIR ranges allows the project to retire
datacenter‑assigned prefixes and operate with a clean, independent addressing plan.
- Service isolation: Migrating Games‑3P from an LXC container to a VM provides better performance
boundaries and operational flexibility.
- Operational hygiene: Housekeeping tasks ensure the environment remains maintainable and
well‑documented.
These changes support FurrIX’s goal of maintaining a clear governance boundary, a stable operational
footprint and a better run vIX while preserving its hobbyist openness.
What this means for members of the exchange
During the rebuild:
- Members may experience extended periods of routing instability or service unavailability during
the maintenance window. These interruptions will be minimized where possible, but the entire
vIX has to come offline for this work to be completed. We are aiming for no more than seven
hours of downtime, but have allotted two days in-case something unexpected happens.
- Member configuration changes will be required:
Member tunnel configurations, peering sessions and addressing assignments will require modification
and our volunteers will email new credentials and profiles soon as we are able to.
- No policy or governance changes:
This is strictly an infrastructure rebuild and does not affect membership, governance or peering policy.
Are exchange operations affected?
Yes — temporarily.
During the rebuild window, routing and service availability will be null as systems are rebuilt
and renumbered. Once the work is complete, normal operations will resume with improved
stability, ease of expansion, better rooted upkeep and clarity.