Don't forget to sign your comments!

Goals

• Availability - Keep the services up and running.
• Integrity - Prevent our Internet presence from being vandalized or subverted.
• Traceability - Track who did what, when. Needed for change control and as a best practice.
• Authenticity - Making sure admins really are who they say they are is important. Otherwise, this is a lower priority for us; most stuff is cleartext and public.
• Privacy - Generally not a priority for us. Most of what we do is public by nature and intent. Still, some things (e.g., passwords) will need to be protected.

In place

Firewall

• Using a local firewall (iptables/netfilter) to lock out most things
• Deny inbound by default
• Allow outbound by default
• Allowed inbound from anywhere:
  • 25/tcp (SMTP)
  • 80/tcp (HTTP)
  • 443/tcp (
  • 53/udp
  • Safe ICMP types (ping, destination unreachable, parameter problem, time expired)
  • SSH -- for specifics, see discussion elsewhere
• Allow 53/tcp from DNS slaves
  • Currently allowed from everywhere because we're lazy :)

SSH

• Public keys are required; passwords disabled
• Users must be a member of the "=sshusers=" group
• Listens on a non-standard TCP port to cut down on log noise

logwatch

This automatically distills the logs, highlighting unusual events. It mails a report every night, and I (BenScott) usually read it on a daily basis.

Suspended

These are practices which we will follow if we ever get more than one person helping out.

SSH

• General root login via SSH not allowed
  • SSH to root allowed with "=forced-commands-only=" to enable backups

sudo

• Any ServerPeople with SSH access have their own user account on the system
• No direct root logins. Login to user account first, then sudo.
• This gives us both enhanced security and traceability (a record of what gets done).
• Avoid "sudo to shell" if at all feasible. If you must, consider using "script" to record the session.

As a policy, we strictly avoid sudo'ing to a shell (e..g., "sudo -i", "sudo /bin/sh"). This is entirely for traceability -- commands done within a sudo shell are not logged. If it must be done, wrap your session in an invocation of script(1) to log the entire shell session.

With this environment of multiple admins from diverse backgrounds who have never worked together before, this is mostly done for record-keeping purposes. Our use of sudo is more about keeping honest people honest then protecting from arbitrary attacks.

Ideas and discussion

auto-start script on sudo to shell

It was suggested that it would be nice if script(1) could be auto-started if a sudo-to-shell was performed. Implementation failed to materialize.

I much prefer autostarting script if 'sudo shell' is run - flexible and easy. Creating time/original user named logfile in standard location would be ideal. (e.g. 2006_02_10_11_53_billm.log) -- DrewVanZandt - 10 Feb 2006

Some things that require shell redirection don't work without this [the ability to sudo to a shell] - however those can be put into a script and run 'sudo script'. But then the script is fungable after the run, so what's been gained is worth debating. -- BillMcGonigle - 08 Feb 2006

I agree with Bruce that we need to be very diligent in keeping track of what we do and keeping strict access and audit controls in place. . So I favor the idea of capturing any root shell sessions in log files. Flat out disabling sudo to a shell is less appealing. It certainly doesn't get us any security (aside from Bill's observation on changing files, sudo can generally be used to defeat itself, unless you lock everyone down to the point where we wouldn't be able to admin the box with it anymore). So our use of sudo will be more about keeping honest people honest then protecting from arbitrary attacks. -- BenScott - 16 Feb 2006

OTP for sudo

OTP (OPIE or some other S/Key) for sudo access. -- MikeLedoux - 13 Feb 2006

Are there any links for those who aren't familar with these? -- BruceDawson - 13 Feb 2006

Both implement your standard challenge/response one time password, and plug in to sudo fairly easily. A quick search didn't find any information that doesn't presume familiarity, I'll see if I can come up with something better tonight. -- MikeLedoux - 13 Feb 2006

I'm a little unsure about how this fits into the risk management equation. What threat are we defending against here? Compromised SSH keys? Won't the overhead of OTP (I assume that's One Time Password; I've never used OPIE or S/Key before) be a large burden for us? -- BenScott - 16 Feb 2006

I think OTP for sudo would be a royal p.i.t.a. I use OTP a lot at work and home, and while it's very secure, and not too painful for basic ssh authentication, it could become seriously cumbersome for sudo access. Additionally, the risk OTP is engineered to protect against might actually be subverted if people were to accidently generate the OTPs on the remote server. -- PaulLussier - 24 Feb 2006

The only thing OTP for sudo would save us from in this case is attacks to extract passwords typed during ssh sessions. Given the decisions we've made regarding that, there's no point in pursuing OTP for sudo any further. -- MikeLedoux - 27 Feb 2006

SSH

Public keys

It has been suggested by several that we require public key authentication and disallow password authentication entirely.

Opinion - I am aware this could cause flames, sorry. Cracking will probably not be from password guessing etc. (especially if on nonstandard port, and we stick to decent passwords e.g. gpw or whatever) - it will be a vulnerability in a service we run. Minimal exposure to the outside world would be good. :-) -- DrewVanZandt - 10 Feb 2006

While I agree that vulnerabilities present a bigger danger, password cracking is a threat. Further, blind, automatic, SSH password guessing attacks are currently active on the Internet. At work we were getting probed not infrequently. Changing the default port foils most of them, but it's still there. At the time time, public keys are pretty easy, generally considered very strong, and actually make things easier if you use ssh-agent. -- BenScott - 16 Feb 2006

Another thing which has not been discussed is the use of OTP for ssh access. This would be significantly more secure than the use of SSH keys. Consider the possibility of a trojaned sshd getting installed on the system. This sshd now has access to the secret passphrase to your key should you ssh from the remote system elsewhere. Using agent-based authentication forwarding and OTP, this risk is mitigated. Any OTP you type is instantly obsolete upon it's successful use. I would find this use of OTP far more palatable than for sudo. -- PaulLussier - 24 Feb 2006

I'm not convinced that OTP for SSH is more secure than SSH keys, provided a) you use a strong passphrase for your SSH key, and b) you only use your SSH key and passphrase directly on a trusted host under your immediate control. Since your passphrase is never sent over the wire to the remote sshd, I'm not sure how a trojaned sshd would ever get that passphrase. Perhaps I am missing something obvious here. -- MikeLedoux - 27 Feb 2006

Access list

Require entries in AllowedUsers in sshd_config. Inconvenient for adding new accounts, but the number of accounts will be very low and we don't have time to deal with getting cracked. -- BillMcGonigle - 11 Feb 2006

Or we could use a Unix group (e.g., "sshusers"). -- BenScott - 16 Feb 2006

Also, no root logins via SSH (public key or otherwise). sudo only. See above. -- BenScott - 16 Feb 2006

Addendum: root login via SSH allowed by public key with "forced-commands-only_ to enable backups via SSH. -- BenScott - 16 Nov 2006

Port number

It has been suggested by several that we use a non-standard port number for SSH (instead of IANA's assignment of 22).

I've been port-scanned for ssh running on non-standard ports. -- BillMcGonigle - 11 Feb 2006

While this won't stop any serious attack, it does stop a lot of the stupid ones, and that counts for something. It also cuts down on log noise, if nothing else. :) -- BenScott - 16 Feb 2006

IP address restrictions

It has been suggested that we restrict SSH to a small set of known IP addresses.

I postulate that restricting ssh access to a small set of IP's is more secure than having it open to the world. It avoids all the script-kiddie issues, for instance. So, there should be arguments here for why we'd want to accept a less-secure method in the face of known attacks. -- BillMcGonigle - 11 Feb 2006

Unfortunately, I do not have a personal static IP address, so this would mean I would not be able to help much (or at all). Active discussion of this issue is presently on the -sysadmin list. -- BenScott - 16 Feb 2006

Likewise, my personal 'net connection has a dynamic IP, so this would severly impair my ability to contribute. -- MikeLedoux - 27 Feb 2006

Other tools

• logsentry/logwatch/etc - alert us to unusual messages (email, maybe SMS) -- BenScott - 16 Feb 2006
• portsentry - identify probes and scans and auto-firewall the source addresses -- BenScott - 16 Feb 2006
  • One note - this would require us to open up any ports we want to sentry in the firewall. Is this something we want to do? -- ColeTuininga - 26 Feb 2006
• System file integrity verification tools:
  • bsign - Corruption & intrusion detection using embedded hashes
  • systraq - monitor your system and warn when system files change
  • tripwire - file and directory integrity checker
    • I can only vouch for tripwire, it works, and it works well. But it require maintenance and produces rather verbose reports which need to be monitored and dealt with. -- PaulLussier - 24 Feb 2006
  • rkhunter/chkrootkit - Looks for known rootkits, and a few other things. -- ColeTuininga - 26 Feb 2006