Wednesday, May 20, 2026
[Incident Report #034][DNS] Inter‑subnet Communication Failure
What Happened?
On May 15th, our upstream data center completed a router upgrade. As an
unintended side effect, the FurrIX subnets located within the data center
were no longer able to reach one another. Because this issue was isolated to
internal data‑center paths, no external member traffic or internet‑facing name server
traffic were affected.
The issue went undetected until May 19th because our monitoring system and
our email system reside on opposite subnets. With inter‑subnet communication
broken, monitoring alerts could not reach us.
We were seeing the following issues:
- Internal service reachability — Some internal services were unreachable from
member connections. - NS2 isolation — Members could not reach NS2.
- Stale zones on NS2 — NS2 could not reach NS1; as a result, its zones
went stale on May 17th. - NMS visibility loss — The Network Management System could not reach devices
on PHY One for accounting and monitoring. - Backup failures — PHY One could not reach the PBS instance on PHY Two,
preventing nightly backups.
What did we do to fix this?
We provided the data center with test results and trace data confirming the inter‑subnet
routing failure. They corrected the configuration on their side, restoring full communication
between PHY One and PHY Two. All internal services, monitoring and backup operations
have returned to normal.
Saturday, April 18, 2026
[Incident Report #032][DNS] SSL Expiry on NS1 and NS2
What Happened?
Our SSL certs for NS1 and NS2 expired earlier today. Currently our process
for handling the updating of SSL certs is not automated and requires our
team to manually install new certs and then reload the servers one after
the other. Usually this is on our internal calendar and is handled three to
four days before EOL. That didn’t happen this time.
We were seeing the following issues:
- Loss of DNS over HTTPS support
- Loss of DNS over TLS support
What did we do to fix this?
- We pulled new certs and updated the cert store
- We reloaded both name servers to restore service
Everything should be operational and peachy again!
Tuesday, March 24, 2026
[Incident Report #031][NET] Connectivity Issues with 2602:F992:EC::/46
Update:
Appears our static route did not get migrated to the new hardware
at the data center. This has been resolved and our services should
be back online momentarily!
What Happened?
Our upstream provider performed some upkeep on the data center on
Mar 20th and since then our routed subnet has not be working properly.
We are looking into this as it affect our IPAM, NMS, PBS and a few other
services that assist us with managing our network.
This is also affects our secondary name server, as it currently does not
have IPv6 service.
We are seeing the following issues:
- No IPv6 connectivity for NS2, affecting private and public resolution request
- No PBS backups for either MFN, FurrIX or Maows.Gay
- No NMS accounting or error tracking
- No status graphs on our website are updating
What are we doing to work on this?
- We have performed internal and external networking checks
- We have reached out to our upstream provider with our findings
At the moment, we will have to wait to hear back from the data center.
Monday, March 2, 2026
[Incident Report #027][DNS] Name Server Attack
What Happened?
From around 0000 to 0400EST last night, our name servers were
hit with a large amount of traffic originating from a data center
in the Netherlands. The traffic was spread across multiple IPv6
subnets and volume was high enough the it saturated the virtual
bridges that our name servers are operating behind. This is the
third attack of this kind that our network seen in the past two weeks.
What damages Resulted?
During the attack, we saw the following issues:
- Accounting service lost contact with both NS1 and NS2
- Shell access was slow or non-responsive
- Legitimate name lookups were being dropped or timed out
- The NMS lost SNMP contact entirely with NS1
- High CPU load on Nardoragon router
What are we doing to deal with this?
Going forward, the following steps will be taken to try and maintain
the usability of our name servers during similar attacks:
- Deploy a separate virtual bridge and network interface for management
- Tightening per subnet rate limits and mask sizing
- Banning AS numbers that partake in attacks like this
- Increasing virtual bridge bandwidth to allow for more throughput
- Redirecting recurring problematic subnets and AS ranges to null
Monday, February 23, 2026
Influx of DNS Traffic
It appears that overnight, our name servers experienced levels of traffic that
our network has not seen before. The request rate reached up to 800+ request
per second incoming and our internal tooling failed to send emails or alert us
via Discord that anything had happened.
It looks like a bunch of VPS instances from a provider over in the Netherlands
started performing mass lookups for a crypto exchange and this traffic was
sustained for quite a while during the night, as our network graphs are showing
and that our 500MB log file limit on both name servers was exhausted.
Our team will be looking into our rate limit configurations and possibly
re-configuring the IDS on our routers to be a little more proactive on alerting
us to conditions like what happened last night.