Monday, September 28, 2026
Transparency Report #016: Dual-Stack Rollout for NTP Service
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What Happened
For the past month, FurrIX has operated its public NTP server (time.furrix.zone) as an
IPv6-only node participating in the NTP Pool Project. To expand time synchronization support
for legacy clients, curious onlookers and external parties, the operations team evaluated
enabling IPv4 access for the service.
This has finally made it through a three step process to be instated as a part of our core
fabric. The first step was a proposal via our Discord from the engineer to our governing
board, which was originally IPv6 only. That proposal got wearily expanded to include IPv4
only after the base IPv6 service went live. The board put the proposal to vote and our
handful of members agreed this was the right way forward. Which led to step three, the
actual IPv6 deployment and reaching out to our upstream about the IPv4 side before we
deployed legacy support.
Because public facing UDP/123 services carry an inherent risk of being targeted by botnets for
NTP reflection/amplification floods, FurrIX proactively opened a ticket with our upstream provider
prior to turning on IPv4 traffic in order to give their engineering and legal departments the chance
to review our operations and provide their stance. We outlined our planned destination IP
(204.12.237.197), NAT topology, daemon choice (chrony), fabric level traffic shaping and rate
limits, as well as other deployment details.
Upstream network engineering reviewed and approved the deployment. They confirmed that
operating a public NTP node on our assigned IPv4 allocation is permitted, provided we understand
the possible risk and noted that upstream automated flow-monitoring is active and may step in
automatically should anything become a problem.
Changes to the Exchange
Following upstream clearance, our volunteers enabled dual-stack functionality for the NTP service:
- Upstream Address Forwarding & Firewall Rules:
Configured a destination NAT rule on our Nardoragon router (OPNsense) forwarding inbound
UDP/123 traffic on 204.12.237.197 to the internal VM instance (10.0.5.3) with
registered firewall rules
- Daemon Configuration (chrony):
Updated chrony.conf on the NTP VM to bind to both its IPv6 GUA (2602:f992:e0:c0::3)
and internal IPv4 interface (10.0.5.3). Active application-layer rate limiting
(ratelimit interval 3 burst 8) and restricted command port bindings (bindcmdaddress ::1)
remain strictly enforced
- Bandwidth & QoS Enforcement:
The 6 Mbps inbound/outbound traffic shaper on the FurrIX fabric remains active for all
NTP traffic. This caps peak packet processing at approximately ~6,500 PPS and helps
limit NTP traffic to limit saturation of exchange links or degradation of routing performance
- DNS Updates:
Added an A record (204.12.237.197) alongside the existing AAAA record for
time.furrix.zone
- Registered IPv4 with NTP Pool
Created a server entry for the IPv4 endpoint
Are Exchange Operations Affected?
Not really, I mean at least in the bad sense that most of you worry about.
The NTP service continued operating without interruption on IPv6 during the rollout and
IPv4 queries began arriving upon DNS propagation. Exchange QoS, peering fabrics and
core routing remain completely unaffected.
Thursday, September 10, 2026
A Note from the Engineer
Hey there,
We know that FurrIX usually communicates with our peers and members through
transparency reports after our volunteers have had time to run over the text a
few times, but we figured it’s probably nice to hear from one of the engineers
every once in a while, directly. (Yes, we have the okay to post without getting
things proof’d!)
The Biggest Change First
FurrIX has been dealing with some performance issue stemming from an arrangement
we made amongst our volunteers in order to keep time and resources spent on the
exchange fair for everyone involved. One of those arrangements was the migration
and hosting of the Games-3P LXC from Marbled Fennec Networks before they closed
operations. Games-3P is a hungry fucking pain in the ass when it comes to CPU time,
but it is one of those things that we can’t just drop because one of our volunteers runs
their Minecraft and DayZ instances on it- so it’s a mild obligation to instead engineer
the problem away to an acceptable level.
Which is just what we have done as of September 8th, 2026.
Going forward, we have modified our hypervisor setup so that the FurrIX fabric and
related vIX services run on CCD0, while the volunteer negotiated services and vIX
test bed run on CCD1. The idea is that our critical services such as routing, NS, NTP
and bits of our tooling will stay isolated in a safe, lower load and predictable side of
our Ryzen 9 3900X; while non-critical, spiky and noisy loads such as the web server,
Games-3P and a few community testing VMs live over on CCD1 and do not affect the
fabric as badly.
Another Service Running!
While futzing around on Reddit, we came across a post about adding more servers to
the NTP Pool Project. NTP has always just kind of been one of those system services
that we knew existed, but never bothered with because it just did its own thing and
never complained. So when we saw this post over on r/HomeLab, we started looking
into NTP and the pool project to see if it was something FurrIX could help with. Turns
out setting up Chrony and jumping into the water isn’t all that rough. So yea, FurrIX
now operates time.furrix.zone on the exchange.
- The RSD-232 Collective
Wednesday, August 19, 2026
Transparency Report #014: PHYTWO (Aedon) OS Hypervisor Upgrade Preparation
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What Happened
As a small team of volunteers, we pour a lot of free time into maintaining the network’s
engine PHYONE- which leaves our secondary server in need of a little TLC that has gotten
a bit long in the tooth.
Our PHYTWO server (Aedon) is overdue for major updates across its entire stack:
- Debian base OS
- Proxmox VE (PVE)
- Proxmox Backup Server (PBS)
These components are now far enough out of date that routine upgrades carry a
higher‑than‑normal risk of failure. To keep the exchange stable, we’re preparing for
both the best‑case and worst‑case outcomes.
Planned Work
We will be attempting a full upgrade of Aedon’s base OS and PVE environment. If everything
doesn’t explode on us, this will be a straightforward process and PHYTWO will return to service
with current software and no major changes. However, because this host is central to our
control‑plane and monitoring stack, we are staging for the possibility that the upgrade may fail
or leave the system in an unrecoverable state.
Pre‑Upgrade Safeguards
To protect the exchange and ensure continuity of essential services, we are taking the
following steps:
- Temporarily suspending NS2 service
NS2 will be offline during the upgrade window to avoid inconsistent state or partial failure scenarios.
- Backing up critical VMs to PHYONE
Any virtual machines required for day‑to‑day operation will be migrated or backed up to PHYONE.
This ensures we can continue running the vIX even if PHYTWO becomes unavailable.
- Preparing for a full rebuild
If the upgrade fails, we will rebuild PHYTWO from scratch, including its network configuration,
Proxmox environment, and PBS instance. This process may take time, but it will be done methodically
to avoid introducing instability into the fabric.
Expected Impact
- NS2 will be offline temporarily.
- Some internal tooling may experience brief interruptions.
- Limited member‑facing and peering services may be affected.
In the event of a failed upgrade, PHYTWO may remain offline for an extended rebuild period due
to our volunteers having outside pressures and commitments, but core vIX operations will continue
on PHYONE. We work on the exchange as time permits and in a method to avoid burnout of our
volunteers during this rather involved project.
Transparency Report #013 — Monitoring Anomalies and RRL Activity
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What Happened
On the afternoon of 18 August 2026, our monitoring system (LibreNMS) began displaying
abnormally large bandwidth spikes for NS2, including inbound and outbound traffic peaks
exceeding hundreds of megabits per second. These values were inconsistent with NS2’s
QoS limited capacity and did not match host‑level counters or resolver behavior.
During the same period, NS2 received a high‑volume PTR sweep from the IPv6 prefix
2a05:f480:2400::/48. Bind’s Response Rate Limiting (RRL) immediately escalated and
began dropping all responses to the prefix. This prevented any outbound meaningful
traffic spikes on the VM and kept resolver load stable.
A review of physical host metrics, VM‑level counters and RRL logs confirmed that
NS2 never transmitted the large volumes of traffic shown in LibreNMS or displayed on
our website graphs. The anomaly was isolated to the monitoring layer and one of our
volunteers pointed out to us that our LibreNMS VM is hitching, slow to respond and
sometimes timing out.
Changes to the Exchange
Our volunteers identified two issues within the monitoring stack:
- Interface Mapping Drift:
LibreNMS occasionally identifies the wrong interfaces for some of our VMs.
We’re currently investigating what is causing this and have an idea, but do
not want to point fingers until we are sure.
- Poller Delays:
The LibreNMS VM showed signs of slow polling and missed intervals. The
internal logs are showing both missed polls and ICMP issues within the
whole exchange, but our testing and probing shows otherwise.
We believe these issues caused the monitoring system to display incorrect bandwidth
values for NS2 and a few other VMs. No changes have been made to the resolver or network
at this time. Operations is preparing remediation options, including a rebuild of the NMS VM since
it still contains data from the project we split from, and will be presenting them to oversight
for consensus before proceeding.
Are Exchange Operations Affected?
No.
NS2 continued operating normally throughout the event. RRL successfully mitigated
the abusive PTR sweep, QoS remained effective and host‑level counters showed
stable, low traffic consistent with expected resolver behavior. The only affected
component was the monitoring system’s visibility layer. Member‑facing, peering and
internal services were not impacted.
Sunday, August 2, 2026
[Incident Report #038][vIX] NS Abuse Impacting Exchange Stability
What are Incident Response Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. During the normal
operations of the exchange, our network and its supporting systems may encounter
operational defects, bugs, failed changes or attacks on our infrastructure. When this
happens, the FurrIX volunteers publish an incident response report to ensure all members
and peers remain informed as to what happened, how it went down and what we did to
recover or resolve the issue. As a hobbyist‑rooted vIX, we aim to keep communication clear,
accessible and practical to the best of our ability.
What Happened?
During August 1st and August 2nd, 2026, the FurrIX vIX experienced a significant
service disruption caused by an unexpected surge of traffic hammering our name
servers. This traffic overwhelmed our exchange fabric, resulting in instability and
degraded performance for multiple services- up to the point of making the exchange
itself unreachable and completely saturated our link to our upstream.
During the events, we have been observing:
- vIX reachability issues — The virtual exchange entered a degraded state
due to excessive inbound NS load.
- NS1/NS2 saturation — Both name servers experienced heavy recursive query
pressure, exhausting logging capacity and reducing responsiveness.
- MA BGP instability — Member BGP sessions dropped from the vIX due to excessive traffic.
- Monitoring failures — NMS monitoring temporarily dropped during peak saturation.
- General service degradation — Web, email and auxiliary services were intermittently unreachable.
This event originated from external recursive DNS traffic and did not involve any FurrIX members.
Upon reviewing logs, traffic alerts and our periodic graphs, it was determined this may be automated
scanner traffic that found and exploited our infrastructure. Our volunteers are currently making
updated procedures and are in the process of updating the vIX configuration to deal with this.
What are we doing to fix this?
During investigation, it was determined that our existing rate‑limit configuration inside Proxmox
did not behave as we originally thought they would and thus did not protect the exchange fabric
from inbound saturation. Furthermore, the traffic that slapped the exchange was rotating through
IPv6 addresses at a rate and pattern that mostly avoided our BIND9 rate limits, which has been
eye opening in itself.
To address this, FurrIX is implementing the following corrective actions:
- Migrating all rate‑limiting from Proxmox to the OpnSense router VMs, where
shaping and policing can be applied correctly at the network demarcation.
- Reworking our NS policy, including tightening recursion rules, implementing
stricter query controls and improving protections for authoritative services.
- Enhancing network monitoring, adding visibility at the router, VM and hypervisor
layers to detect similar events earlier; including active alerting to our Discord.
Exchange operations may remain unstable while volunteers complete these changes and validate
the new configuration.
Current Status
As of this post, the exchange fabric has recovered and all core FurrIX vIX services are reachable.
Additional hardening work is ongoing and further updates will be posted once our volunteers have
completed all the ongoing work.