Our servers use utilities such as fail2ban or denyhosts that look for repeated failed login attempts, and once found they direct the firewall service to ban the originating IP addresses.  However this technique fails when the attack is distributed among thousands of compromised “zombie” computers that are doing the bidding of a malicious hacker.

Our log files correctly diagnosed each attempt from an individual IP address, but a new attempt was immediately started from a different IP address.  We were clearly looking at an attack coordinated from a single unknown source.

There are several ways to reduce or thwart these attacks, including:

• never allow remote root logins (but attacks still occur on non-root-user names)
• have a chroot jail shell in case an attack on a non-root account succeeds
• changing SSH service to use a non-obvious port (such as port 5022 instead of 22)
• deactivate password authentication and rely exclusively on authentication keys
• restrict allowed IP address by country of origin
• only allow certain IP addresses or ranges of addresses to have access

We chose to implement more than one of these solutions, and I wanted to share some techniques we used for our implementation.

For the last rule that only allows certain IP addresses, I wanted to start with a list of valid IP addresses used in the last month.  The following script extracts these IP numbers from our SSH log file, sorts them alphabetically, and then removes duplicates.  Note that this server uses Fedora – you may need to tweak it for other linux distributions.

root# fgrep "Accepted" /var/log/secure* | awk '{print $11}' | sort | uniq
166.77.6.4
205.232.34.1
67.255.5.155
...

The IP addresses from the above script should be added to the file /etc/hosts.allow in the following format:

# hosts.allow   This file contains access rules which are used to
#               allow or deny connections to network services that
#               either use the tcp_wrappers library or that have been
#               started through a tcp_wrappers-enabled xinetd.
#
#               See 'man 5 hosts_options' and 'man 5 hosts_access'
#               for information on rule syntax.
#               See 'man tcpd' for information on tcp_wrappers

# allow local addresses
all: 127.0.0.1
all: 192.168.1.*

# valid IP addresses gathered June 2010
all: 166.77.6.4
all: 205.232.34.1
all: 67.255.5.155
...

Now disallow all other IP addresses for SSH by editing the file /etc/hosts.deny:

# hosts.deny    This file contains access rules which are used to
#               deny connections to network services that either use
#               the tcp_wrappers library or that have been
#               started through a tcp_wrappers-enabled xinetd.
#
#               The rules in this file can also be set up in
#               /etc/hosts.allow with a 'deny' option instead.
#
#               See 'man 5 hosts_options' and 'man 5 hosts_access'
#               for information on rule syntax.
#               See 'man tcpd' for information on tcp_wrappers
#
# The portmap line is redundant, but it is left to remind you that
# the new secure portmap uses hosts.deny and hosts.allow.  In particular
# you should know that NFS uses portmap!

# deny SSH service except for IP numbers in /etc/hosts.allow file
sshd: all

Restart your SSH service, and your server should now be a bit more secure against distributed dictionary attacks:

root# service sshd restart

An Internet search using keywords from the other mentioned solutions above will teach you how to change SSH port, disallow password authentication, etc.

A firewall is used to only allow traffic to the outside world on a few of the TCP/UDP ports.  We obviously have to allow web and e-mail users access to the server, but almost all other ports can be closed to prevent intrusion attempts.  On our newest servers we even prevent FTP and Telnet access, since those protocols rely on unencrypted packets which are easier to intercept and hijack.

Every day we have perhaps dozens of “dicitionary” attacks that try to gain e-mail or user account access.  A dictionary attack picks a user (for example “root” or “john”) and then goes through a long, long list of possible passwords.  We use two packages Fail2Ban and DenyHosts that monitor our log files looking for dictionary attacks; if found, the originating computer is banned from accessing our servers.

When we develop online shopping solutions, we choose to not store credit card numbers online.  We securely pass this information to the credit card processing vendor, and then we only record the order information and the payment confirmation number.  For some web sites with user accounts, we encrypt the user account passwords, therefore gaining access to our user password list would still not result in someone gaining access to their online account.

Some of our hosting customers are concerned about unencrypted web traffic.  We occasionally add a feature that automatically forwards a web page inquiry from non-SSL to SSL mode, which means it forward to a page starting with “https://” thus all traffic is encrypted between our server and each web browser client.

We also have logging records and constant monitoring to help us detect intrusion attempts and help us implement even better security measures.  “Tripwire” software can also alert us when certain files are modified.

Do these basic measures above make us impervious to hackers?  Alas, no.  On two occasions in the last five years we have had hackers penetrate one of our servers.  However no damage was done and we patched those specific holes quickly.  Security is a cat-and-mouse game, and we strive to stay one step ahead.