Friday, September 25, 2026
Incident Report #040: Layer One Failure
What are Incident Response Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. During the normal
operations of the exchange, our network and its supporting systems may encounter
operational defects, bugs, failed changes or attacks on our infrastructure. When this
happens, the FurrIX volunteers publish an incident response report to ensure all members
and peers remain informed as to what happened, how it went down and what we did to
recover or resolve the issue. As a hobbyist‑rooted vIX, we aim to keep communication clear,
accessible and practical to the best of our ability.
What Happened
On Friday, September 25, 2026, the FurrIX vIX experienced an extended outage, rendering the
core network unreachable from approximately 07:00 to 16:30 EST. Members and peers were
unable to establish or maintain connections to the exchange, resulting in a full disruption of
peering and routing services.
Given similar historical occurrences this year, our volunteers suspected a layer one issue
and contacted the data center onsite staff. Upon investigation, a physical Ethernet connection
defect was identified. The cable was reterminated, which successfully restored network
connectivity.
Current Status
As of this post, the exchange fabric has recovered and all core FurrIX vIX services are reachable.
Thursday, September 3, 2026
[Curtain Pull] Today FurrIX Learned: Spelling Matters
Our lead engineer has spent six months loosely trying to chase down what we
thought was a bug in our vhost configs for Apache. Well, today, after budging
and letting a second set of eyes look over the web server configs- FurrIX has
learned that spelling really matters.
A little fun fact about this situation: marbledfennec.net was misspelled in its own
vhost for half a year and none of us noticed, even while trying to debug the host.
Apache did exactly what it was told, not what we meant. People wanted
marbledfennec.net, but the vhost was named marbledfennc.net, so Apache2 just
decided “fuck it, you get 232.gay, enjoy!”
Everything works now. Please do not perceive us.
Thursday, August 20, 2026
Incident Report #039: Backup Server Upgrade Fail
What are Incident Response Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. During the normal
operations of the exchange, our network and its supporting systems may encounter
operational defects, bugs, failed changes or attacks on our infrastructure. When this
happens, the FurrIX volunteers publish an incident response report to ensure all members
and peers remain informed as to what happened, how it went down and what we did to
recover or resolve the issue. As a hobbyist‑rooted vIX, we aim to keep communication clear,
accessible and practical to the best of our ability.
What Happened
Our backup server had fallen significantly out of date across its software stack and during
an attempted upgrade. The update process broke roughly halfway through, leaving the system
in an unstable state and temporarily removing our ability to perform backups for the night.
Despite the failed upgrade, our volunteers successfully preserved all datastores before any data
loss could occur.
Upon assessing the severity of the failed upgrade, it became clear that attempting to salvage the
partially‑updated system would be too risky to FurrIX’s data. Our volunteers therefore used their
emergency governance override to authorize a full rebuild of the backup server. This decision allowed
us to move forward with a clean installation of the latest Proxmox Backup Server (PBS) version rather
than attempting to repair a heavily broken environment. Once the new PBS installation was in place,
the volunteers imported the preserved datastores and verified that the system could read all existing
backup data without any errors.
Points of Note
- Backups were unavailable for one night during the failure and rebuild window.
- No CT or VM backups appear to have been lost.
- All on‑hand data has been verified as readable on both PHYONE and PHYTWO.
- Normal backup operations have resumed under the rebuilt PBS environment.
The exchange avoided data loss and now has a fully updated and stable backup server. As always,
we appreciate the patience and understanding of our members and peers while we maintain and
improve the infrastructure that keeps FurrIX running.
Wednesday, August 19, 2026
Transparency Report #013 — Monitoring Anomalies and RRL Activity
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What Happened
On the afternoon of 18 August 2026, our monitoring system (LibreNMS) began displaying
abnormally large bandwidth spikes for NS2, including inbound and outbound traffic peaks
exceeding hundreds of megabits per second. These values were inconsistent with NS2’s
QoS limited capacity and did not match host‑level counters or resolver behavior.
During the same period, NS2 received a high‑volume PTR sweep from the IPv6 prefix
2a05:f480:2400::/48. Bind’s Response Rate Limiting (RRL) immediately escalated and
began dropping all responses to the prefix. This prevented any outbound meaningful
traffic spikes on the VM and kept resolver load stable.
A review of physical host metrics, VM‑level counters and RRL logs confirmed that
NS2 never transmitted the large volumes of traffic shown in LibreNMS or displayed on
our website graphs. The anomaly was isolated to the monitoring layer and one of our
volunteers pointed out to us that our LibreNMS VM is hitching, slow to respond and
sometimes timing out.
Changes to the Exchange
Our volunteers identified two issues within the monitoring stack:
- Interface Mapping Drift:
LibreNMS occasionally identifies the wrong interfaces for some of our VMs.
We’re currently investigating what is causing this and have an idea, but do
not want to point fingers until we are sure.
- Poller Delays:
The LibreNMS VM showed signs of slow polling and missed intervals. The
internal logs are showing both missed polls and ICMP issues within the
whole exchange, but our testing and probing shows otherwise.
We believe these issues caused the monitoring system to display incorrect bandwidth
values for NS2 and a few other VMs. No changes have been made to the resolver or network
at this time. Operations is preparing remediation options, including a rebuild of the NMS VM since
it still contains data from the project we split from, and will be presenting them to oversight
for consensus before proceeding.
Are Exchange Operations Affected?
No.
NS2 continued operating normally throughout the event. RRL successfully mitigated
the abusive PTR sweep, QoS remained effective and host‑level counters showed
stable, low traffic consistent with expected resolver behavior. The only affected
component was the monitoring system’s visibility layer. Member‑facing, peering and
internal services were not impacted.
Sunday, August 2, 2026
[Incident Report #038][vIX] NS Abuse Impacting Exchange Stability
What are Incident Response Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. During the normal
operations of the exchange, our network and its supporting systems may encounter
operational defects, bugs, failed changes or attacks on our infrastructure. When this
happens, the FurrIX volunteers publish an incident response report to ensure all members
and peers remain informed as to what happened, how it went down and what we did to
recover or resolve the issue. As a hobbyist‑rooted vIX, we aim to keep communication clear,
accessible and practical to the best of our ability.
What Happened?
During August 1st and August 2nd, 2026, the FurrIX vIX experienced a significant
service disruption caused by an unexpected surge of traffic hammering our name
servers. This traffic overwhelmed our exchange fabric, resulting in instability and
degraded performance for multiple services- up to the point of making the exchange
itself unreachable and completely saturated our link to our upstream.
During the events, we have been observing:
- vIX reachability issues — The virtual exchange entered a degraded state
due to excessive inbound NS load.
- NS1/NS2 saturation — Both name servers experienced heavy recursive query
pressure, exhausting logging capacity and reducing responsiveness.
- MA BGP instability — Member BGP sessions dropped from the vIX due to excessive traffic.
- Monitoring failures — NMS monitoring temporarily dropped during peak saturation.
- General service degradation — Web, email and auxiliary services were intermittently unreachable.
This event originated from external recursive DNS traffic and did not involve any FurrIX members.
Upon reviewing logs, traffic alerts and our periodic graphs, it was determined this may be automated
scanner traffic that found and exploited our infrastructure. Our volunteers are currently making
updated procedures and are in the process of updating the vIX configuration to deal with this.
What are we doing to fix this?
During investigation, it was determined that our existing rate‑limit configuration inside Proxmox
did not behave as we originally thought they would and thus did not protect the exchange fabric
from inbound saturation. Furthermore, the traffic that slapped the exchange was rotating through
IPv6 addresses at a rate and pattern that mostly avoided our BIND9 rate limits, which has been
eye opening in itself.
To address this, FurrIX is implementing the following corrective actions:
- Migrating all rate‑limiting from Proxmox to the OpnSense router VMs, where
shaping and policing can be applied correctly at the network demarcation.
- Reworking our NS policy, including tightening recursion rules, implementing
stricter query controls and improving protections for authoritative services.
- Enhancing network monitoring, adding visibility at the router, VM and hypervisor
layers to detect similar events earlier; including active alerting to our Discord.
Exchange operations may remain unstable while volunteers complete these changes and validate
the new configuration.
Current Status
As of this post, the exchange fabric has recovered and all core FurrIX vIX services are reachable.
Additional hardening work is ongoing and further updates will be posted once our volunteers have
completed all the ongoing work.