Archive for the ‘Security’ Category

Website and App Maintenance Agreements

Thursday, February 16th, 2012

Maintenance Agreements – They are — They are not

Websites and apps are not commodities and the purchasing of them is not a one-night-stand. Websites and apps are born into dynamic environments. Their full value is realized when they are maintained current with those environments. The developer relationship established during the initial development is part of the purchased value. Economies accrue when a structured relationship is maintained.

A Maintenance Agreement provides the budget to prevent obsolescence. The sad news is that a new website or app is becoming obsolete even as it is built. All three sectors of the technology advance rapidly.

  • New display and interaction devices replace the old at increasing rates.
  • There is daily industry news about greater power in databases and middleware.
  • Methods-of-use, ways of thinking about these technologies, sprint into our lives.

The users of any valuable website or app will chafe immediately when it does not take advantage of this or that new device, or some new way of searching, or some social opportunity. Staying atop these concerns is the noble use of a maintenance agreement.

A Maintenance Agreement budget also should be provided for installing important updates to the underlying software. The price should be small, maybe $250 per event, maybe two or three events per year. Security updates in particular must be done.

A Maintenance Agreement provides a management structure for all the above. There is a prioritized wish list, a clear line of communication, and a non-intrusive billing arrangement.

The Maintenance Agreement budget is not for hosting the app and ensuring its connectivity. These services are budgeted under Hosting Agreements.

GORGES would be embarrassed if Maintenance Agreement budgets were spent mostly on fixing the app. In some cases this is appropriate use, but there should be very little of this. Websites and apps are warranted for thirty days after going live and after major upgrades. Clients should test the app thoroughly before and during that period to take advantage of the warranty.

The best websites and apps pay their own way. Allocating a portion of the return for maintenance extends their lives and empowers their constituencies.

Staging servers

Monday, August 22nd, 2011

At GORGES we prefer to set up a staging server or website for our bigger projects.  I would even call it a requirement since it can be used by the customer and our quality assurance staff to review both progress and the release candidate before publishing a website to a production site.

Usually developers work directly on their workstations or laptops, and a development environment has both web and database services running similar to a production server. Ideally a development environment should have the same versions of services as the production server, for example PHP 5.2 or Ruby 1.9.  On more than one occasion I have been bitten by writing MySQL 4.3 or 5 code, but the production server has an older version of MySQL without sub-query support so the queries had to be rewritten.

Usually these development services are customized for development, for example the database does not have to be hardware-optimized and the web service can have debugging tools enabled.  I prefer to turn on all warning messages, so any identified problems that are not severe enough to halt page serving will be shown either in a log file or onscreen.

Just as important with having a staging server is to have clean and well-structured publishing scripts.  If you are relying on a manual process to upload files using FTP, then it is possible to skip an updated file by mistake.  Publishing scripts come in all sorts of flavors and approaches – we have both “pull” methods that use server-side scripts, and “push” scripts that execute on the developer’s computer and automatically upload files to the server.  The push scripts are built on Capistrano, which although it is based on ruby we sometimes use at Gorges for non-ruby websites.

We routinely protect our staging servers from prying eyes and search engines by using HTAuth username and passwords.  Once a website is scanned by a Google, Bing, or other web crawler then it is out of your control.

Having a staging server doesn’t mean much unless it is actually used to test new features and look for bugs.  Quality assurance is important, and I have yet to meet a developer that produces bug-free code or see a unit testing system that covers 100% of a web application feature set.

Not covered in this post is the issue of database migrations between developer environment and a staging server, and also from a staging server to a production machine.  These can be tricky, but as with all challenges the proper planning will guide you to the best solution.

After the application or update goes live to the production server most staging servers are continued as “sand boxes” to support additional updates and testing.

worked in academia, corporate research labs and several technology startup companies prior to GORGES. His expertise is software architecture, database development, and system administration. Matt brings GORGES over 25 years experience developing fast and robust software on a multitude of platforms and languages.

Distributed Dictionary Attack Solutions

Saturday, June 26th, 2010

We have had the misfortune of having attempted distributed dictionary attacks on our Linux servers.  A dictionary attack uses a long list of common usernames and passwords trying to find a way to gain a foothold and eventually root access of a password-protected server.

Our servers use utilities such as fail2ban or denyhosts that look for repeated failed login attempts, and once found they direct the firewall service to ban the originating IP addresses.  However this technique fails when the attack is distributed among thousands of compromised “zombie” computers that are doing the bidding of a malicious hacker.

Our log files correctly diagnosed each attempt from an individual IP address, but a new attempt was immediately started from a different IP address.  We were clearly looking at an attack coordinated from a single unknown source.

There are several ways to reduce or thwart these attacks, including:

  • never allow remote root logins (but attacks still occur on non-root-user names)
  • have a chroot jail shell in case an attack on a non-root account succeeds
  • changing SSH service to use a non-obvious port (such as port 5022 instead of 22)
  • deactivate password authentication and rely exclusively on authentication keys
  • restrict allowed IP address by country of origin
  • only allow certain IP addresses or ranges of addresses to have access

We chose to implement more than one of these solutions, and I wanted to share some techniques we used for our implementation.

For the last rule that only allows certain IP addresses, I wanted to start with a list of valid IP addresses used in the last month.  The following script extracts these IP numbers from our SSH log file, sorts them alphabetically, and then removes duplicates.  Note that this server uses Fedora – you may need to tweak it for other linux distributions.

root# fgrep "Accepted" /var/log/secure* | awk '{print $11}' | sort | uniq
166.77.6.4
205.232.34.1
67.255.5.155
...

The IP addresses from the above script should be added to the file /etc/hosts.allow in the following format:

# hosts.allow   This file contains access rules which are used to
#               allow or deny connections to network services that
#               either use the tcp_wrappers library or that have been
#               started through a tcp_wrappers-enabled xinetd.
#
#               See 'man 5 hosts_options' and 'man 5 hosts_access'
#               for information on rule syntax.
#               See 'man tcpd' for information on tcp_wrappers

# allow local addresses
all: 127.0.0.1
all: 192.168.1.*

# valid IP addresses gathered June 2010
all: 166.77.6.4
all: 205.232.34.1
all: 67.255.5.155
...

Now disallow all other IP addresses for SSH by editing the file /etc/hosts.deny:

# hosts.deny    This file contains access rules which are used to
#               deny connections to network services that either use
#               the tcp_wrappers library or that have been
#               started through a tcp_wrappers-enabled xinetd.
#
#               The rules in this file can also be set up in
#               /etc/hosts.allow with a 'deny' option instead.
#
#               See 'man 5 hosts_options' and 'man 5 hosts_access'
#               for information on rule syntax.
#               See 'man tcpd' for information on tcp_wrappers
#
# The portmap line is redundant, but it is left to remind you that
# the new secure portmap uses hosts.deny and hosts.allow.  In particular
# you should know that NFS uses portmap!

# deny SSH service except for IP numbers in /etc/hosts.allow file
sshd: all

Restart your SSH service, and your server should now be a bit more secure against distributed dictionary attacks:

root# service sshd restart

An Internet search using keywords from the other mentioned solutions above will teach you how to change SSH port, disallow password authentication, etc.

worked in academia, corporate research labs and several technology startup companies prior to GORGES. His expertise is software architecture, database development, and system administration. Matt brings GORGES over 25 years experience developing fast and robust software on a multitude of platforms and languages.

Securing Linux Web Servers

Monday, July 20th, 2009

We are often asked by our software development and hosting customers how we secure our servers.  We have several layers of security protection, and this blog posting will mention some that we implement.

A firewall is used to only allow traffic to the outside world on a few of the TCP/UDP ports.  We obviously have to allow web and e-mail users access to the server, but almost all other ports can be closed to prevent intrusion attempts.  On our newest servers we even prevent FTP and Telnet access, since those protocols rely on unencrypted packets which are easier to intercept and hijack.

Every day we have perhaps dozens of “dicitionary” attacks that try to gain e-mail or user account access.  A dictionary attack picks a user (for example “root” or “john”) and then goes through a long, long list of possible passwords.  We use two packages Fail2Ban and DenyHosts that monitor our log files looking for dictionary attacks; if found, the originating computer is banned from accessing our servers.

When we develop online shopping solutions, we choose to not store credit card numbers online.  We securely pass this information to the credit card processing vendor, and then we only record the order information and the payment confirmation number.  For some web sites with user accounts, we encrypt the user account passwords, therefore gaining access to our user password list would still not result in someone gaining access to their online account.

Some of our hosting customers are concerned about unencrypted web traffic.  We occasionally add a feature that automatically forwards a web page inquiry from non-SSL to SSL mode, which means it forward to a page starting with “https://” thus all traffic is encrypted between our server and each web browser client.

We also have logging records and constant monitoring to help us detect intrusion attempts and help us implement even better security measures.  “Tripwire” software can also alert us when certain files are modified.

Do these basic measures above make us impervious to hackers?  Alas, no.  On two occasions in the last five years we have had hackers penetrate one of our servers.  However no damage was done and we patched those specific holes quickly.  Security is a cat-and-mouse game, and we strive to stay one step ahead.

worked in academia, corporate research labs and several technology startup companies prior to GORGES. His expertise is software architecture, database development, and system administration. Matt brings GORGES over 25 years experience developing fast and robust software on a multitude of platforms and languages.
©2013 GORGES - All rights reserved
where programming meets design and lives happily ever after