Posts Tagged ‘security’

Distributed Dictionary Attack Solutions

Saturday, June 26th, 2010

We have had the misfortune of having attempted distributed dictionary attacks on our Linux servers.  A dictionary attack uses a long list of common usernames and passwords trying to find a way to gain a foothold and eventually root access of a password-protected server.

Our servers use utilities such as fail2ban or denyhosts that look for repeated failed login attempts, and once found they direct the firewall service to ban the originating IP addresses.  However this technique fails when the attack is distributed among thousands of compromised “zombie” computers that are doing the bidding of a malicious hacker.

Our log files correctly diagnosed each attempt from an individual IP address, but a new attempt was immediately started from a different IP address.  We were clearly looking at an attack coordinated from a single unknown source.

There are several ways to reduce or thwart these attacks, including:

  • never allow remote root logins (but attacks still occur on non-root-user names)
  • have a chroot jail shell in case an attack on a non-root account succeeds
  • changing SSH service to use a non-obvious port (such as port 5022 instead of 22)
  • deactivate password authentication and rely exclusively on authentication keys
  • restrict allowed IP address by country of origin
  • only allow certain IP addresses or ranges of addresses to have access

We chose to implement more than one of these solutions, and I wanted to share some techniques we used for our implementation.

For the last rule that only allows certain IP addresses, I wanted to start with a list of valid IP addresses used in the last month.  The following script extracts these IP numbers from our SSH log file, sorts them alphabetically, and then removes duplicates.  Note that this server uses Fedora – you may need to tweak it for other linux distributions.

root# fgrep "Accepted" /var/log/secure* | awk '{print $11}' | sort | uniq
166.77.6.4
205.232.34.1
67.255.5.155
...

The IP addresses from the above script should be added to the file /etc/hosts.allow in the following format:

# hosts.allow   This file contains access rules which are used to
#               allow or deny connections to network services that
#               either use the tcp_wrappers library or that have been
#               started through a tcp_wrappers-enabled xinetd.
#
#               See 'man 5 hosts_options' and 'man 5 hosts_access'
#               for information on rule syntax.
#               See 'man tcpd' for information on tcp_wrappers

# allow local addresses
all: 127.0.0.1
all: 192.168.1.*

# valid IP addresses gathered June 2010
all: 166.77.6.4
all: 205.232.34.1
all: 67.255.5.155
...

Now disallow all other IP addresses for SSH by editing the file /etc/hosts.deny:

# hosts.deny    This file contains access rules which are used to
#               deny connections to network services that either use
#               the tcp_wrappers library or that have been
#               started through a tcp_wrappers-enabled xinetd.
#
#               The rules in this file can also be set up in
#               /etc/hosts.allow with a 'deny' option instead.
#
#               See 'man 5 hosts_options' and 'man 5 hosts_access'
#               for information on rule syntax.
#               See 'man tcpd' for information on tcp_wrappers
#
# The portmap line is redundant, but it is left to remind you that
# the new secure portmap uses hosts.deny and hosts.allow.  In particular
# you should know that NFS uses portmap!

# deny SSH service except for IP numbers in /etc/hosts.allow file
sshd: all

Restart your SSH service, and your server should now be a bit more secure against distributed dictionary attacks:

root# service sshd restart

An Internet search using keywords from the other mentioned solutions above will teach you how to change SSH port, disallow password authentication, etc.

The distributed dictionary attack has been going on for five straight days. By my informal estimates, there have been about 30,000 attempts to break in on our coyglen server.

After weighing options, I decided to restrict access by IP number. I was a bit gun-shy after Friday night’s mistake of trying to restrict by country of origin and subsequent onsite hard-reboot.

Below I have pasted the unique usernames and IP addresses used for valid SSH logins on coyglen in the last 30 days. I used the IP#s to create a list of valid computers and added them to the /etc/hosts.allow file.

After restarting the SSH service, I am delighted to see many connection-refusal messages.

If a customer cannot get access using SSH or SFTP (note: insecure FTP is not supported on this server), then have them send you an e-mail message. We will open up their e-mail headers and look at the IP#, then add it to the allowed list.

I have not applied this solution to our other servers, but will if needed. If someone controls a “zombie” network, then it is trivial to direct attacks like this trying to find insecure hosting servers. I expect more of these to happen in the future.

Matt

[root@coyglen log]# fgrep “Accepted” secure* | awk ‘{print $9}’ | sort | uniq

cogentex

csg

csg2

dgreen

elimilice

gorges

gorges-blog

gorges-test

healthytest

myteambook

napoli

netway-dev

ngrown

paramount

pmpauthors

qbuilder

retweeter

root

royaltystat

sales.wwm

stablecom.sta

stlnps

userfinity

valmontgod

[root@coyglen log]# fgrep “Accepted” secure* | awk ‘{print $11}’ | sort | uniq

166.137.136.169

166.77.6.4

184.74.139.251

192.168.1.3

208.105.225.93

24.58.62.167

24.58.67.78

24.59.191.200

38.108.102.253

64.57.177.68

64.57.178.15

66.180.184.17

67.244.55.108

67.246.70.140

67.255.18.108

67.255.3.244

67.255.5.155

69.205.224.88

70.94.56.13

71.181.154.195

71.181.238.165

71.183.96.7

72.43.91.102

74.66.27.169

Matt Clark worked in academia, corporate research labs and several technology startup companies prior to GORGES. His expertise is software architecture, database development, and system administration. Matt brings GORGES over 25 years experience developing fast and robust software on a multitude of platforms and languages.

Passwords that are Used by Groups

Monday, April 12th, 2010

There are good reasons to not make group passwords, but then there are times when it is the best method, such as during web application development. Then, there is the problem of how to generate these strong passwords and how to communicate about them.

Strong passwords are essential to data security. Passwords are easy enough to produce, but they are not as effective as they should be because human foibles lead to compromises. Humans find strong passwords hard to remember so they write them down. They find them hard to say so they send them in email to coworkers.

These problems can be overcome and it can even be fun.

Make the strong password by making it along with a mnemonic. Like this:

Which bear ate up my donut holes
Wb8^mdOs

Now we can refer to this as the “donut” password without giving much away and it is memorable.

We can also make a family of passwords:

Bears do not care for those expensive holes
B!c4t$Os

Is it useful to have two related passwords that may be called “the bear question” and “the bear answer”? Say, for the admin password and the staff password?

Assigning words to the shift characters on the number keys is the trick here, but not too tricky. “^” is top or up or above. “@” is at or each. “$” is money or cash or dollars ……
You get it.

Now we have pass words that can be remembered without using word combinations, i.e., thefudge, or word bastardizations, i.e., es$enti@l, either of which is easily cracked.

Now we also have passwords that can be named even in the presence of strangers without giving them away, you could just say, “Try the bear answer.”

You may also use these ideas to make your own private passwords.

While you are thinking about this, it is also a great way to generate “family passwords.”

Jane never cared for asparagus
jnc4asparagus or jnc4agus

Have fun!

Securing Linux Web Servers

Monday, July 20th, 2009

We are often asked by our software development and hosting customers how we secure our servers.  We have several layers of security protection, and this blog posting will mention some that we implement.

A firewall is used to only allow traffic to the outside world on a few of the TCP/UDP ports.  We obviously have to allow web and e-mail users access to the server, but almost all other ports can be closed to prevent intrusion attempts.  On our newest servers we even prevent FTP and Telnet access, since those protocols rely on unencrypted packets which are easier to intercept and hijack.

Every day we have perhaps dozens of “dicitionary” attacks that try to gain e-mail or user account access.  A dictionary attack picks a user (for example “root” or “john”) and then goes through a long, long list of possible passwords.  We use two packages Fail2Ban and DenyHosts that monitor our log files looking for dictionary attacks; if found, the originating computer is banned from accessing our servers.

When we develop online shopping solutions, we choose to not store credit card numbers online.  We securely pass this information to the credit card processing vendor, and then we only record the order information and the payment confirmation number.  For some web sites with user accounts, we encrypt the user account passwords, therefore gaining access to our user password list would still not result in someone gaining access to their online account.

Some of our hosting customers are concerned about unencrypted web traffic.  We occasionally add a feature that automatically forwards a web page inquiry from non-SSL to SSL mode, which means it forward to a page starting with “https://” thus all traffic is encrypted between our server and each web browser client.

We also have logging records and constant monitoring to help us detect intrusion attempts and help us implement even better security measures.  “Tripwire” software can also alert us when certain files are modified.

Do these basic measures above make us impervious to hackers?  Alas, no.  On two occasions in the last five years we have had hackers penetrate one of our servers.  However no damage was done and we patched those specific holes quickly.  Security is a cat-and-mouse game, and we strive to stay one step ahead.

Matt Clark worked in academia, corporate research labs and several technology startup companies prior to GORGES. His expertise is software architecture, database development, and system administration. Matt brings GORGES over 25 years experience developing fast and robust software on a multitude of platforms and languages.