Finding the cause of constant high CPU on a security log forwarder

A log forwarder is a server that collects a client's logs, from their firewalls, computers and cloud services, and passes them on to us so we can watch for security issues. One client, an enterprise asset management software and consulting company, had a forwarder running close to full CPU. In business hours its logs fell up to 2 hours behind, so our monitoring of that client lagged with them.

The other engineers on the team had all looked into it and couldn't find the cause. The busiest service on the forwarder logged no errors while it spun, so its own logs gave nothing to go on.

Part of the answer was outside the server, in the client's firewall: the device that checks traffic going in and out of their network. A firewall that inspects traffic closely can end a connection it doesn't like by sending a reset, a message that tells both ends the connection is over.

After each restart, the service that collects the client's cloud logs sat idle, then climbed a whole CPU core at a time, with flat memory and no errors. Of 36 servers on the same version, only this one did it.

Packet captures showed the client's firewall sending fake resets into that service's cloud connections: 44 in 5 minutes, each with a network fingerprint that didn't match the real traffic. A bug in Filebeat's Azure input meant it didn't recover cleanly when its connections were cut this way. The client changed a firewall rule so the storage connection skipped inspection, and I asked for the same on the second connection.