Prevent IP address spoofing using Reverse path filtering

This post is about system administration and has nothing to do with Python. Then why post this here? Because I believe there many like me who run one or more web servers and sometime run into these problems.

In the previous post I wrote that my ISPConfig Debian server was subject to port scanning, etc. and that it appeared that 95% of all requests came from China unless ... these IP addresses were spoofed of course.

I never looked into this, there were problems sometimes but they lasted only for a short time. More recent, it became very frequent, like multiple times a day. I had to dig in and it appeared source address verification was NOT enabled for my server. This also means that maybe completely normal, spoofed IP addresses are (temporarily) blocked by fail2ban, the opposite of what we want!

Now why do these installation tutorials never tell you this? Or is it TL;DR? After I enabled source address verification, the number of port scanner IP addresses dropped dramatically, and I did not see any IP address from China anymore. This means that hackers, or should I call them script kiddies, abuse IP addresses from Chinese companies, for their criminal activities.

Anyway, below I write down the first steps of what I did, everything is also on the internet of course, see links at the end of this post.

My server is a VPS connected to the internet via the routers of my hoster. It runs Debian 11 (Bullseye), ISPConfig, and is connected to the internet using a single ipv4 IP address. This means that the instructions below are for ipv4 only. And you can also skip all this if you have a hoster who already implemented all these protections for you.

Prevent IP address spoofing using Reverse path filtering

There are several types of IP address spoofing. First step is drop packets from IP addresses that are not routable. Source Address verification is available on Linux servers as Reverse path filtering. This is controlled by kernel parameter 'rp_filter'. You can set kernel parameters in the file:

/etc/sysctl.conf

You can set them using a command, example:

sysctl -w 'net.ipv4.ip_forward=1'

To check the current settings, we can use the 'sysctl' command and filter the parameter. Example:

sysctl -ar '\.rp_filter'

Result:

net.ipv4.conf.all.rp_filter = 0
...
net.ipv4.conf.default.rp_filter = 0
...
net.ipv4.conf.eth0.rp_filter = 0
...

These values can be found in files the directory /proc/sys, for example:

cat /proc/sys/net/ipv4/conf/eth0/rp_filter

Note the 'all', 'default', and 'eth0'. 'default' is used for new interfaces.

Important: These values are sometimes AND-ed, somestimes OR-ed, sometimes the maximum value is used. This is specified in the document '/proc/sys/net/ipv4/* Variables', see links below. If you do not follow this document, some parameters will not work, or you may disable or enable a setting accidently!

While we are at it, we also set some other kernel parameters, see the document '4.18. Securing network access', see links below. I added the following lines to the file:

/etc/sysctl.conf

And I changed only the settings for the interface 'eth0'!

...

# Ignore ICMP broadcasts
net/ipv4/icmp_echo_ignore_broadcasts = 1

# Do not accept ICMP redirects (prevent MITM attacks)
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0 
net.ipv4.conf.eth0.accept_redirects = 0

# Do not send ICMP redirects (we are not a router)
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.eth0.send_redirects = 0

# Turn on Source Address Verification to prevent some spoofing attacks
net.ipv4.conf.eth0.rp_filter = 1

# Log Martians
net.ipv4.conf.eth0.log_martians = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Do not accept IP source route packets (we are not a router)
net.ipv4.conf.eth0.accept_source_route = 0

Here are the notes on the used kernel parameters:

accept_redirects - BOOLEAN
	Accept ICMP redirect messages.
	accept_redirects for the interface will be enabled if:
	- both conf/{all,interface}/accept_redirects are TRUE in the case
	  forwarding for the interface is enabled
	or
	- at least one of conf/{all,interface}/accept_redirects is TRUE in the
	  case forwarding for the interface is disabled
	accept_redirects for the interface will be disabled otherwise
	default TRUE (host)
		FALSE (router)
		
send_redirects - BOOLEAN
	Send redirects, if router.
	send_redirects for the interface will be enabled if at least one of
	conf/{all,interface}/send_redirects is set to TRUE,
	it will be disabled otherwise
	Default: TRUE

rp_filter - INTEGER
	0 - No source validation.
	1 - Strict mode as defined in RFC3704 Strict Reverse Path
	    Each incoming packet is tested against the FIB and if the interface
	    is not the best reverse path the packet check will fail.
	    By default failed packets are discarded.
	2 - Loose mode as defined in RFC3704 Loose Reverse Path
	    Each incoming packet's source address is also tested against the FIB
	    and if the source address is not reachable via any interface
	    the packet check will fail.

	Current recommended practice in RFC3704 is to enable strict mode
	to prevent IP spoofing from DDos attacks. If using asymmetric routing
	or other complicated routing, then loose mode is recommended.

	The max value from conf/{all,interface}/rp_filter is used
	when doing source validation on the {interface}.

	Default value is 0. Note that some distributions enable it
	in startup scripts.

log_martians - BOOLEAN
	Log packets with impossible addresses to kernel log.
	log_martians for the interface will be enabled if at least one of
	conf/{all,interface}/log_martians is set to TRUE,
	it will be disabled otherwise

accept_source_route - BOOLEAN
	Accept packets with SRR option.
	conf/all/accept_source_route must also be set to TRUE to accept packets
	with SRR option on the interface
	default TRUE (router)
		FALSE (host)

After we made the changes, we load them using:

sysctl -p

We have also enabled logging of martian packets i.e. packets with a source address which is obviously wrong, it is not possible to route back to that address. To see these loglines:

grep martian /var/log/messages

Result:

...
Jun 14 15:52:53 <server> kernel: [    3.742674] IPv4: martian source <dst-ip-address> from <src-ip-address>, on dev eth0
Jun 14 15:52:53 <server> kernel: [    3.743462] IPv4: martian source <dst-ip-address> from <src-ip-address>, on dev eth0
Jun 14 15:52:53 <server> kernel: [    3.743468] IPv4: martian source <dst-ip-address> from <src-ip-address>, on dev eth0

Next steps

The above is just a first step in hardening your server against all kinds of attacks. For example, in the document 'DDoS Protection With IPtables: The Ultimate Guide', see links below, much more measures are described.

Check your logs

Other things you should do is check your logs! Bad or misconfigured services can cause all sort of problems, including making your server more vulnerable. How can these things happen? Well, for example you upgrade to a new version of Debian. This may break some services or leave them in a bad state, even if your server appears to run fine.

Checking /var/log/syslog, I found lines like (I replaced the IP addresses etc.):

...
Jun 14 10:23:33 server123 named[743]: REFUSED unexpected RCODE resolving 'iwnde.ro/MX/IN': 198.51.100.167#53
Jun 14 10:39:04 server123 named[743]: REFUSED unexpected RCODE resolving '203.0.113.183.in-addr.arpa/PTR/IN': 198.51.100.12#53
...

Never saw these before. Can be spammers sending email to my server and mark them as coming from bogus domains. It looks like their number significantly decreased after turning on Reverse path filtering. Have to look into this one of the next days.

Simple monitor script

I wanted to see what's happening minute-by-minute and created this tiny script that outputs some basic information to a file using some Linux command line utilities:

#!/bin/bash
# syslpcm.sh
# Append system information to logfile
# - l oad averages
# - p rocess count
# - c onnection count
# - m emory available / swap free
# 
# add these two lines to (root) cron to run every minute:
# # syslpcm
# * * * * * /root/syslpcm.sh

LOGDIR=/root
LOGFILE=${LOGDIR}/syslcpm.log

load_averages=$(uptime | awk -F'[a-z]:' '{ print $2}')
processes_count=$(ps -e | wc -l)
connections_count=$(netstat -an | wc -l)
mem_available=$(awk '/MemAvailable/ { printf "%.3f \n", $2/1024/1024 }' /proc/meminfo)
swap_free=$(awk '/SwapFree/ { printf "%.3f \n", $2/1024/1024 }' /proc/meminfo)
dt=`date "+%Y-%m-%d %H:%M:%S"`
progress_message="${dt} load avgs: ${load_averages} procs: ${processes_count} cons: ${connections_count} mem_avail: ${mem_available} swap_free: ${swap_free}"
echo "${progress_message}" >> "${LOGFILE}"

After you added this script to CRON the logfile will look like:

2023-06-14 17:14:01 load avgs:  1.07, 0.98, 1.04 procs: 447 cons: 1252 mem_avail: 1.744  swap_free: 3.596 
2023-06-14 17:15:17 load avgs:  1.28, 1.05, 1.06 procs: 451 cons: 1342 mem_avail: 1.731  swap_free: 3.430 
2023-06-14 17:16:01 load avgs:  12.59, 4.66, 2.32 procs: 446 cons: 1326 mem_avail: 1.900  swap_free: 3.376 
2023-06-14 17:17:01 load avgs:  5.76, 4.13, 2.28 procs: 442 cons: 1272 mem_avail: 2.004  swap_free: 3.323 
2023-06-14 17:18:01 load avgs:  3.29, 3.69, 2.24 procs: 443 cons: 1333 mem_avail: 2.110  swap_free: 3.175 
2023-06-14 17:19:02 load avgs:  2.13, 3.25, 2.18 procs: 437 cons: 1381 mem_avail: 2.348  swap_free: 3.160 
2023-06-14 17:20:01 load avgs:  1.13, 2.75, 2.07 procs: 442 cons: 1297 mem_avail: 2.338  swap_free: 3.160 
2023-06-14 17:21:01 load avgs:  0.51, 2.28, 1.95 procs: 441 cons: 1301 mem_avail: 2.333  swap_free: 3.138 
2023-06-14 17:22:02 load avgs:  0.50, 1.96, 1.86 procs: 440 cons: 1323 mem_avail: 2.233  swap_free: 3.139 

Summary

This took time. Hardening your server against attacks has become extremely important, the internet is NOT a safe place. I know that system administration is very difficult and respect the people who do this. But here I am, a programmer running a VPS, that is connected to the internet, himself, so I need to know some of these things. Not a waste of time to learn this but essential knowledge when you have servers connected to the internet.

Using the rp_filter kernel parameter setting to enable Reverse path filtering was not difficult, and I also changed a few parameters because the server is not a router. The result was astonishing, almost no more port scanners. In addition, by looking again in the logs I noticed something that was not there before, have to look into this.

Note that if someone really wants to take your server down, there is not much you can do except to move to a DDoS protected network. Creating is hard, destroying is easy.

Links / credits

/proc/sys/net/ipv4/* Variables
https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

/proc/sys/net/ipv4/* Variables (other link)
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/Documentation/networking/ip-sysctl.txt?h=linux-4.15.y

40 Linux Server Hardening Security Tips [2023 edition]
https://www.cyberciti.biz/tips/linux-security.html

DDoS Deflate
https://github.com/jgmdev/ddos-deflate

DDoS Protection With IPtables: The Ultimate Guide
https://javapipe.com/blog/iptables-ddos-protection

Don’t Get Swept Away: The Most Common DDoS Attacks are SYN and UDP Floods
https://www.masterdc.com/blog/how-ddos-attack-work-types-of-ddos-syn-udp-floods

How to Disable ICMP Redirects in Linux for security (Redhat,Debian,Ubuntu,SuSe tested)
https://www.itsyourip.com/Security/how-to-disable-icmp-redirects-in-linux-for-security-redhatdebianubuntususe-tested/

How to interpret Linux martian source messages
https://www.thegeekdiary.com/how-to-interpret-linux-martian-source-messages

How to protect my Ubuntu server ?
https://medium.com/@d_danailov/how-to-protect-my-ubuntu-server-6a2ecc056722

IP Spoofing
https://www.imperva.com/learn/ddos/ip-spoofing

Ispconfig 3 - DDOS attack mitigation
https://forum.howtoforge.com/threads/ispconfig-3-ddos-attack-mitigation.82316

rp_filter and LPIC-3 Linux Security
https://www.theurbanpenguin.com/rp_filter-and-lpic-3-linux-security

Securing Debian Manual - 4.18. Securing network access
https://www.debian.org/doc/manuals/securing-debian-manual/network-secure.en.html

What is IP Spoofing? Types of IP Spoofing
https://www.interserver.net/tips/kb/ip-spoofing-types-ip-spoofing

What is IP Spoofing? Types of IP Spoofing
https://www.interserver.net/tips/kb/ip-spoofing-types-ip-spoofing

What is the difference between "all", "default" and "eth*" in /proc/sys/net/ipv[46]/conf/?
https://unix.stackexchange.com/questions/90443/what-is-the-difference-between-all-default-and-eth-in-proc-sys-net-ipv