Tuesday, September 29, 2026
Transparency Report #017: NTP Service Stats!
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What Happened
The volunteers at FurrIX have been working on finalizing the NTP deployment and getting
it to a point that we do not have to keep a hawk’s eye on what is does. This means we
have gotten internally tooling working for us and webpage goodies for you all!
Also, we have done our best to extend the life of PHYTWO and our LibreNMS instance
a little bit longer, but we are running out of useful life on that box.
Changes to the Exchange
LibreNMS:
- RRDCached:
Our volunteers setup rrdcached on the LibreNMS server to give us a little bit of a break
when it comes to the performance of our monitoring stack. Let’s look at the reality, the
PHYTWO control plane and backup server is old. It needs to be retired and something
else put in it’s place, but budget and small little tweaks are keeping it alive…somehow.
- Board Discussion:
The FurrIX board is in conversation about retiring the current PHYTWO box and moving
it to a purpose built, much newer box that would be backed by a fibre connection that
actually exceeds what FurrIX itself is capable of. This would mean that one of the volunteers
would have physical control of that box, so some new policy and guidelines have to be made.
NTP service:
- LibreNMS Enrollment:
The NTP VM is enrolled in our network monitoring, meaning our volunteers will now get
Discord and email alerts if things start to lag, go over network thresholds or drift.
- Status Graphs:
You guys love stats. We love stats. Everyone loves stats.
The FurrIX website now shows averages of the packets per second in and out for the
NTP service for one hour, six hours, one day and one week. Enjoy, nerds!
Are Exchange Operations Affected?
No. These changes were done early enough in the day that our operations were not
bothered in the slightest and the fabric didn’t even notice the few things that did get
restarted service wise.
Monday, September 28, 2026
Transparency Report #016: Dual-Stack Rollout for NTP Service
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What Happened
For the past month, FurrIX has operated its public NTP server (time.furrix.zone) as an
IPv6-only node participating in the NTP Pool Project. To expand time synchronization support
for legacy clients, curious onlookers and external parties, the operations team evaluated
enabling IPv4 access for the service.
This has finally made it through a three step process to be instated as a part of our core
fabric. The first step was a proposal via our Discord from the engineer to our governing
board, which was originally IPv6 only. That proposal got wearily expanded to include IPv4
only after the base IPv6 service went live. The board put the proposal to vote and our
handful of members agreed this was the right way forward. Which led to step three, the
actual IPv6 deployment and reaching out to our upstream about the IPv4 side before we
deployed legacy support.
Because public facing UDP/123 services carry an inherent risk of being targeted by botnets for
NTP reflection/amplification floods, FurrIX proactively opened a ticket with our upstream provider
prior to turning on IPv4 traffic in order to give their engineering and legal departments the chance
to review our operations and provide their stance. We outlined our planned destination IP
(204.12.237.197), NAT topology, daemon choice (chrony), fabric level traffic shaping and rate
limits, as well as other deployment details.
Upstream network engineering reviewed and approved the deployment. They confirmed that
operating a public NTP node on our assigned IPv4 allocation is permitted, provided we understand
the possible risk and noted that upstream automated flow-monitoring is active and may step in
automatically should anything become a problem.
Changes to the Exchange
Following upstream clearance, our volunteers enabled dual-stack functionality for the NTP service:
- Upstream Address Forwarding & Firewall Rules:
Configured a destination NAT rule on our Nardoragon router (OPNsense) forwarding inbound
UDP/123 traffic on 204.12.237.197 to the internal VM instance (10.0.5.3) with
registered firewall rules
- Daemon Configuration (chrony):
Updated chrony.conf on the NTP VM to bind to both its IPv6 GUA (2602:f992:e0:c0::3)
and internal IPv4 interface (10.0.5.3). Active application-layer rate limiting
(ratelimit interval 3 burst 8) and restricted command port bindings (bindcmdaddress ::1)
remain strictly enforced
- Bandwidth & QoS Enforcement:
The 6 Mbps inbound/outbound traffic shaper on the FurrIX fabric remains active for all
NTP traffic. This caps peak packet processing at approximately ~6,500 PPS and helps
limit NTP traffic to limit saturation of exchange links or degradation of routing performance
- DNS Updates:
Added an A record (204.12.237.197) alongside the existing AAAA record for
time.furrix.zone
- Registered IPv4 with NTP Pool
Created a server entry for the IPv4 endpoint
Are Exchange Operations Affected?
Not really, I mean at least in the bad sense that most of you worry about.
The NTP service continued operating without interruption on IPv6 during the rollout and
IPv4 queries began arriving upon DNS propagation. Exchange QoS, peering fabrics and
core routing remain completely unaffected.
Friday, September 25, 2026
Incident Report #040: Layer One Failure
What are Incident Response Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. During the normal
operations of the exchange, our network and its supporting systems may encounter
operational defects, bugs, failed changes or attacks on our infrastructure. When this
happens, the FurrIX volunteers publish an incident response report to ensure all members
and peers remain informed as to what happened, how it went down and what we did to
recover or resolve the issue. As a hobbyist‑rooted vIX, we aim to keep communication clear,
accessible and practical to the best of our ability.
What Happened
On Friday, September 25, 2026, the FurrIX vIX experienced an extended outage, rendering the
core network unreachable from approximately 07:00 to 16:30 EST. Members and peers were
unable to establish or maintain connections to the exchange, resulting in a full disruption of
peering and routing services.
Given similar historical occurrences this year, our volunteers suspected a layer one issue
and contacted the data center onsite staff. Upon investigation, a physical Ethernet connection
defect was identified. The cable was reterminated, which successfully restored network
connectivity.
Current Status
As of this post, the exchange fabric has recovered and all core FurrIX vIX services are reachable.
Friday, September 18, 2026
Transparency Report #015: Routine Upkeep
What are Transparency Reports?
As a community‑operated and governed virtual internet exchange, FurrIX maintains
a commitment to open and honest communication with its members. From time to
time, operational work may occur that affects the exchange or its supporting infrastructure.
When this happens, the FurrIX operations team publishes a transparency report to
ensure all members remain informed. As a hobbyist‑rooted vIX, we aim to keep
communication clear, accessible and practical to the best of our ability.
What Happened
Not too much to report on this time around, just some upkeep on the fabric for
the parts that have not been automated just yet. Mainly that NS2 does not have
a direct some upkeep path for when its SSL certs expire. We should be working
on that in the next week or so.
Impact on the Exchange
- NS2 should be working over DoH and DoT again.
Thursday, September 10, 2026
A Note from the Engineer
Hey there,
We know that FurrIX usually communicates with our peers and members through
transparency reports after our volunteers have had time to run over the text a
few times, but we figured it’s probably nice to hear from one of the engineers
every once in a while, directly. (Yes, we have the okay to post without getting
things proof’d!)
The Biggest Change First
FurrIX has been dealing with some performance issue stemming from an arrangement
we made amongst our volunteers in order to keep time and resources spent on the
exchange fair for everyone involved. One of those arrangements was the migration
and hosting of the Games-3P LXC from Marbled Fennec Networks before they closed
operations. Games-3P is a hungry fucking pain in the ass when it comes to CPU time,
but it is one of those things that we can’t just drop because one of our volunteers runs
their Minecraft and DayZ instances on it- so it’s a mild obligation to instead engineer
the problem away to an acceptable level.
Which is just what we have done as of September 8th, 2026.
Going forward, we have modified our hypervisor setup so that the FurrIX fabric and
related vIX services run on CCD0, while the volunteer negotiated services and vIX
test bed run on CCD1. The idea is that our critical services such as routing, NS, NTP
and bits of our tooling will stay isolated in a safe, lower load and predictable side of
our Ryzen 9 3900X; while non-critical, spiky and noisy loads such as the web server,
Games-3P and a few community testing VMs live over on CCD1 and do not affect the
fabric as badly.
Another Service Running!
While futzing around on Reddit, we came across a post about adding more servers to
the NTP Pool Project. NTP has always just kind of been one of those system services
that we knew existed, but never bothered with because it just did its own thing and
never complained. So when we saw this post over on r/HomeLab, we started looking
into NTP and the pool project to see if it was something FurrIX could help with. Turns
out setting up Chrony and jumping into the water isn’t all that rough. So yea, FurrIX
now operates time.furrix.zone on the exchange.
- The RSD-232 Collective