I was experiencing a worrying issue on one of my servers recently. It’s a web server that hosts some low traffic WordPress blogs for some customers. Recently, the disk I/O chart for the server showed sudden rise in disk read, as shown in the graph below.
Investigating the issue
To investigate the issue, I connected to the server via SSH and ran the command pidstat -dl 20 which displays processes’ disk usage statistics, showing the full command for each process, and accumulating data for 20 seconds. This is very useful to capture sudden disk usage that is hard to capture otherwise.
The top process for disk usage turned out to be fail2ban .. Unfortunately I didn’t capture a screenshot at the time, but it was 2 or maybe 3 orders of magnitude higher than any other process!
fail2ban is a very common tool for blocking attackers. It works by monitoring various log files for suspicious access attempts, and blocks the attacker’s IP. fail2ban keeps a log of attacks it detected in the file /var/log/auth.log. By investigating that file (using tail and grep … etc), it turned out there were tremendous amounts of attacks to the SSH port. This seems to have been keeping fail2ban really busy parsing logs and banning huge numbers of IPs, causing the high disk load.
Fixing the issue, “outside the box”!
I decided to make use of DigitaOcean’s cloud firewall feature. It’s a free-of-charge feature that allows you to configure an external firewall protecting your server, offloading all the load from fail2ban to an external resource away from your own server (outside the box!)
The trick is that I don’t need to allow access to the SSH port (port 22) from any IPs except my own IP. Unlike web traffic, which should be allowed from all IPs so that users can visit your website, the SSH port is typically accessed from a single IP address in most cases.
So, I created 2 firewall rules to allow incoming (inbound) traffic from all IPs to ports 80 and 443 (which are the ports needed to allow users to visit the websites hosted on the server). And a firewall rule to allow inbound traffic to port 22 (SSH port) from a single IP address, my IP. See the screenshot below.
My IP is not a static one. So, it might change every couple of days, denying me access to the SSH port of the server. But this is not a concern at all. All I need to do is open my DigitalOcean dashboard > Networking > Firewalls > Select the firewall I applied to the server, and edit the SSH rule to change the IP address to the new one. I can then access the server easily.
I know the above is an additional step that I didn’t have to do before, but it’s a trivial step that takes less than a minute. Actually, it can even be automated via DigitalOcean API but I’m too lazy to do it at the moment 😀
So, did this fix the issue?
Here is the disk I/O chart one day after the above fix.
This not only dramatically reduced disk I/O, but also overall server load:
For such a simple fix, this is an impressive gain. Cloud firewall is a free service provided by DigitalOcean. There is no reason not to take advantage of it to secure your server and greatly improve its performance.
DigitalOcean provides a very intuitive interface that makes it very easy to launch your servers in a few minutes. Compared to Amazon AWS and Google GCP, it’s much easier to understand and use. And pricing is also lower and much easier to understand. Sign up to DigitalOcean using this link to receive $100 of free credit to experiment with their awesome service.