NullRabbit Logo

No One Really Knows What's Hitting Their Infrastructure

TL;DR: Your servers are being scanned right now. Linux doesn't log it. Cloud doesn't flag it. Some setups do have visibility - but it's probably not yours. Unless you've deliberately bolted something on, you're blind.

There's a weird assumption a lot of engineers still carry: "If someone scans my server, I'll see it in the logs." Or worse: "Scans don't really matter - they're illegal anyway, no one does them."

I made this assumption myself for nearly 20 years. In the 2000s I had my own servers in my kitchen, in datacentres. We used iptables. Through ignorance mostly, I just assumed people weren't going to get in and hoped for the best.

Years later I'm using private Kubernetes clusters, Google's firewalls, Cloudflare - the works. But I still didn't really know who was running what against my infrastructure. That realisation was a milestone.

Like when your phone starts beeping at 3am because someone's DDoSing you and your house is essentially on fire, but it's fine because you have Cloudflare - only to discover many DDoS attacks are too small for them to flag but still big enough to take your dashboards down.

The Experiment

Recently I was running scans on my own Google Cloud infrastructure and got my account suspended. The message said I was running scans from my GCE instances, which wasn't true. But I thought it was actually quite cool they'd flagged something amongst all that traffic, so I started digging in.

I assumed - again - that because it was Linux, monitoring this would be straightforward.

So I fired up an Ubuntu VM on DigitalOcean and scanned the hell out of it. Got the big guns out. Hammered it with thousands of SYNs, UDP probes, every port imaginable.

I watched dmesg -w, journalctl -f, /var/log/syslog, even the old iptables -j LOG trick. I ran a SYN flood on it.

Nothing. Not a single line. Zero evidence anything happened.

Here's the interesting part: the iptables LOG counters increased. The kernel definitely saw the packets - it just didn't bother telling me about them. Not "blocked". Not "filtered". Not "malformed". Just normal behaviour.

Nothing much to be seen in the following video. We're running quite an aggressive masscan on the Ubuntu server we're running in Digital Ocean.

Why This Matters

Scanning isn't a pentest - it's reconnaissance. The first thing attackers do to work out what's open. Once they know, they look at service versions, probe deeper. Then one day, I guarantee you, they will be in.

And you will have no idea.

I know this from the other side too. I do Due Diligence work - asking companies getting acquired about their infrastructure. Question one is always "have you had a data breach?" No one has ever said yes. But I've found emails, addresses, entire databases sitting publicly accessible. Once, someone's family photos.

Over the last few months I've looked at hundreds of public DeFi validators and was horrified by the results. I was pretty transparent about it, spoke to several of the teams involved. Not one of them had noticed anything. And SSH? Still open on port 22 everywhere. Not everyone, but far more than you'd expect.

The Modern Visibility Problem

Back in the mid-2000s, everything was iptables and a perimeter firewall. We thought visibility came for free. Then cloud arrived - hypervisors, virtual NICs, upstream filtering, Cloudflare, Kubernetes. Every layer adds protection, but none of them give you actual packet visibility.

The modern stack makes it unbelievably easy to run infrastructure with zero awareness of what's touching it. Unless you deliberately add something, your servers are completely silent.

In my demo, the only time I finally saw the traffic was after adding a real packet logger. Not an IDS, not a SIEM - literally just forcing the kernel to hand packet metadata to something that writes it down.

Some Tools That Actually Show External Scans

Lightweight Options

tcpdump - Perfect for quick checks. Zero config, instant truth.

sudo tcpdump -ni any tcp

You'll see masscan/Nmap immediately. Not for long-term use, but unbeatable for "is anything even arriving?"

iptables NFLOG + ulogd2 - The smallest real solution. NFLOG hands packet metadata to userspace without touching dmesg.

sudo iptables -I INPUT 1 -j NFLOG --nflog-prefix "SCAN "
sudo apt install ulogd2
sudo journalctl -fu ulogd2

conntrack - Shows connection attempts and state transitions. Great for spotting brute-force scans.

sudo conntrack -E

Heavier Options

Suricata - Free, noisy, powerful. Good for behavioural signatures and proper detection.

Zeek - Deep network analysis. Turns traffic into structured logs you can analyse over time.

eBPF/XDP hooks - The modern approach. Catches packets before the kernel processes them. Ultra-low overhead, perfect for custom logging or enforcement.

The Takeaway

Your servers are getting scanned constantly. The stack won't tell you. Cloud won't tell you. Linux definitely won't tell you.

Even with the tools available, it's borderline impossible to keep track. More on this soon and what you can do!

validatorssecurityopen-sourcetoolsdepinnode-ops