next_inactive up previous

OpenSSH Tips and Tricks for GOLUG

Written and presented by Kevin Korb

1 What is OpenSSH?

OpenSSH is both client and server programs (ssh and sshd) for a secure method of obtaining a shell on a remote system. Some other features of OpenSSH (taken from their web site):

2 Securing the server side (sshd)

The OpenSSH server (sshd) is very secure straight out of the box. The biggest threat by default is users with weak passwords especially if they are OS default accounts. However, there are many options in the sshd_config file to enhance and tailor security to your needs.

2.1 PermitRootLogin

This setting controls if and how root is allowed to log in via the ssh (after authentication of course). The possible settings for this option are:

2.2 PasswordAuthentication

This setting allows you to disable password authentication completely. The sshd will not even prompt for passwords if other authentication methods fail. This is a good way to force cryptographic authentication (keys) and to prevent bots from ever guessing a password (or even trying). Note that you may or may not need to turn off UsePAM depending on your OS configuration. Test and make sure. Also note that the user must not have a locked password as sshd refuses logins to accounts with locked passwords.

2.3 AllowAgentForwarding, AllowTcpForwarding, GatewayPorts, X11Forwarding

These are all settings that allow you to enable or disable these additional features of ssh. If for instance you wanted to make an account that is only to be used for sftp access then simply denying access to the shell is not enough. You wouldn't want them running a GUI application or using your server as a proxy to probe your internal network either.

2.4 Match

This is a feature of the sshd_config file that allows you to specify settings that are different depending on who is logging in or where they are connecting from. For instance you can specify that PasswordAuthentication is enabled whenever someone connects from your local LAN's IP range but not from anywhere else. You can also force a user or group of users into a specific command instead of a shell using ForcedCommand. A common example is setting up a user who can only run sftp with something like:

Match Group sftponly

  ForceCommand internal-sftp

2.5 Port

This setting allows you to specify an alternate TCP port number for sshd to listen on (the default is 22). This will greatly reduce but not eliminate the number of bots that will attack your sshd trying to guess passwords. There are also things you can do within whatever firewalling you are using to deal with these problems in addition to or instead of changing the port number.

3 Host based authentication

Host based authentication is a quick way to allow password-less logins for many users across several systems without having to setup individual keys for each user. Instead the hosts authenticate each other using host keys then any account that exists on both systems can ssh from one to the other without passwords. This is a common configuration on corporate LANs and even home networks where one person is managing the authentication for all users. It is much easier to simply verify that you trust that the system attempting to log in is one that you trust to have already authenticated the user than it is to generate and maintain keys for each user. To enable this authentication method you must turn on HostbasedAuthentication in both sshd_config and ssh_config files on all systems, add all hosts public keys to /etc/ssh/ssh_known_hosts, and add the host names to /etc/hosts.equiv.

4 Key based authentication

Key based authentication is when a user has a private/public key pair within their account usually in ~/.ssh/id_* and ~/.ssh/id_*.pub where the * denotes which encryption algorithm it uses. The public half of that key pair can be appended to ~/.ssh/authorized_keys on other hosts allowing that user to ssh to those other hosts without typing in the password for the account on the remote host. These key files may be created with or without passphrases to decrypt them. To setup key based authentication from ``client'' to ``server'':

  1. On client run: 'ssh-keygen -t type' (type can be rsa, dsa, or ecdsa). This will create a key pair in your ~/.ssh directory and will allow you to set a passphrase for that key. If you don't want a passphrase just hit enter when prompted.
  2. Either paste the contents of ~/.ssh/id_*.pub on the client into ~/.ssh/authorized_keys on the server or as a shortcut you can run: 'ssh-copy-id server'.
  3. Verify the permissions (on both systems). The ~/.ssh directory must be 0700, the ~/.ssh/* files must all be 0600, and every parent directory of ~/.ssh must not have group or other write permissions and must be owned by either the user or root.
  4. Running 'ssh server' on the client should now log you into the server without a password. Note that if you put a passphrase on your key you will be prompted for that but that is a different thing and there is a way around it (see the agent section).

5 Tightening key based authentication

Once you have key authentication working there are ways you can tighten up the security around a key in addition to encrypting the key file with a passphrase. All of these settings go into the ~/.ssh/authorized_keys file and are prepended to the line containing the key that they are restricting.

5.1 from=''pattern''

This allows you to specify that an ssh key is only to be excepted from the specified hosts or networks. The pattern can be a comma separated list of host names, IP addresses, CIDR blocks, and can use wild cards.

5.2 command=''command line''

This allows you to specify that an ssh key is only allowed to run the specified command. The key will not be allowed to run a shell. If the user attempts to log into the shell or specify a command this command will be run instead. Any user supplied command or arguments will be available as $SSH_ORIGINAL_COMMAND so that the command being forced can act as a wrapper allowing multiple user supplied commands or arguments.

5.3 There are other options that restrict other features such as port forwarding. They are documented in the AUTHORIZED_KEYS FILE FORMAT section of man sshd.

6 Setting up an agent to aid in key based authentication

An agent is a daemon program that holds the unencrypted version of your ssh key in memory and allows the ssh client to read it via a socket instead of the standard files. The agent that comes with OpenSSH is called ssh-agent and is generally run as a parent process to the desktop environment of a client system. This way any child process of the desktop environment (such as a terminal window) will have access to the agent. Once the agent is loaded keys can be added to it with the ssh-add command. If the key being added has a passphrase there will be a prompt for it. There are 3 things that an agent allows you to do that are above what you can do with plain key files:

6.1 Prevent repeated typing of passphrases

When you first load a passphrased key into your agent you will be prompted for the passphrase. The agent then remembers the unencrypted version of the key. This means when you authenticate via the agent instead of the key file you don't have to type in your passphrase again and again. Until the key is either removed from the agent or the agent terminates your key will behave as if it does not have a passphrase on it.

6.2 Allow loading of keys from other systems

Once you have an agent running you can load a key file into it from another system. If you run 'ssh -A user@host ssh-add .ssh/id_dsa' you will add user@host's private dsa key to your agent allowing you to authenticate to systems using that key even though it is not stored on your local system at all.

6.3 Allowing use of your key on remote systems

Once you have an agent running you can allow your logins on other systems to authenticate using it. If you were to run 'ssh -A server' on your client you could then inside of that ssh session run 'ssh server2' the keys in your agent would be used for authentication even if your account on ``server'' has no keys installed.

7 Setting up a client side configuration file to reduce typing

Now that you have learned some complex features of OpenSSH you may find yourself routinely running ssh with many options. You will likely find that you often use the same options to connect to a host every time (specifying user name and port number are probably the most common). These settings can all be specified in your ~/.ssh/config file on your client either as global defaults or host name specific defaults. Here is a simple example:

Host *

  Compression no

Host *.workdomain

  User username

  Compression yes

Host paranoidguy

  User username

  Port 1022
This example does many things...

8 Connection sharing

It is possible to configure ssh to make only a single TCP connection and authentication to a remote host while having many multi-plexed ssh sessions running through it. These sessions can be additional shells, port forwarding, file transfers, or anything else that OpenSSH supports. Because additional connections do not require TCP handshaking or authentication they are established much faster. Implementing this requires minor entries in your ~/.ssh/config file like this:

Host *

  ControlMaster auto

  ControlPath /tmp/ssh-localusername-%r@%h:%p
Change the localusername string to match your user name to prevent any collisions with other users. Once you have this setup your first ssh connection to a host will establish the tcp connection and the authentication. Until that session ends any further connection attempts will use the same connection. If you attempt to close the session while other connections are using it it will simply wait for them to finish. Note that if you are experimenting with authentication this should be disabled since your connection attempts will bypass the authentication stage.

9 Connecting to a remote internal system via an external gateway

A common requirement is to allow an external user to connect to an internal system with a private IP address via ssh. Obviously extra steps need to be taken to make this work. If there is only one internal system then the most common solution is to configure the router to forward tcp/22 connections from the Internet to the internal system. Then the external user can connect via ssh just like normal and not even know that the routing is involved. However, when there are multiple internal systems things get more interesting. Obviously you could use the 'ssh -A' trick mentioned in the agent section to ssh to a single internal system then ssh from there to the correct internal system but that isn't very efficient or automatic. There are two ways to handle this situation in a nice automated way...

9.1 A single gateway server

If you setup port forwarding on the router to forward ssh connections to a single system (we will call it ``gateway'') then that gateway system can be used as a proxy server to allow connections to any internal hosts. The client would setup a ~/.ssh/config entry that looks like this:

Host internalhost

  ProxyCommand ssh gateway -W %h:%p
The user can now simply type 'ssh internalhost' and be logged directly into the internal host even though the user's computer has no connectivity to the internal host or even DNS resolution for that name. Of course you must be able to authenticate to both gateway and internalhost.

Note that this method does make 2 separate ssh connections. If there is a lot of data or if gateway's CPU is weak then the next method may be faster though more complex to setup.

9.2 Multiple ports forwarded to each internal system

You can also set the router up to forward a series of TCP ports to a series of internal systems. For instance tcp/22001 may be forwarded to tcp/22 on the first system, tcp/22002 to tcp/22 on the second system, etc. The client would then configure something like:

Host internalhostname001

  Port 22001

  HostName router

  HostKeyAlias internalhostname001
This would allow the user to run 'ssh internalhostname001' and ssh would know that instead it should connect to ``router'' on tcp/22001 and that it should use the known_hosts entry for ``internalhostname001''.

10 scp and sftp

These are both programs that come with OpenSSH and enable file transfers over an ssh connection. The scp program is designed to operate like the cp command except that either the source or the target can be in the syntax of user@host:path instead of just a local path. The sftp program is designed to act like the ftp program with an interactive command line interface. Note that many GUI file transfer programs (like FileZilla) also support sftp internally.

11 sshfs

Sshfs is a fuse based file system that allows you to mount a remote directory using sftp. It is not actually part of OpenSSH but can be found at

12 Port forwarding (predefined specific ports)

OpenSSH has the ability to tunnel other TCP based protocols through the ssh stream providing them with both encryption and an alternative path into a remote system or network. This is often used to secure traffic that is normally unencrypted without the extra work of setting up a full VPN. Unlike a VPN it does not use additional IP addresses or routing just TCP ports. Tunneling is done using the -L (or -R to go the other direction) parameter and the syntax is -L localport:remotehost:remoteport. Note that the remote host name does not have to be the same host name that ssh is connecting to however the tunnel will be between endpoints leaving that final connection in the clear. An example for tunneling imap4 would be 'ssh -L 10143:mail.domain.tld:143 domain.tld'. This example would establish an ssh connection to domain.tld and from there an imap4 connection to mail.domain.tld (presumably on the same machine or at least the same network). The email client would then connect to localhost:10143 for a secure connection across the Internet rather than mail.domain.tld:143. Note that you can specify more than one -L per ssh command to tunnel multiple ports at the same time.

12.1 Managing ssh tunneling processes

Since the ssh connection running the tunnel does not require a shell to be open to the remote system (in fact the user may not even have authorization for a shell on the remote system) there are multiple ways you can manage the ssh process. Here are two of the popular ones...

12.1.1 Permenant connection

If you want to have a permanent connection that is simply always listening on your local port then you probably want to run ssh without a remote shell and under some form of service management. If you add -T and -N to your ssh parameters then ssh will setup the port forwarding and do nothing else. This ssh command can then be put under some form of process management such as daemontools, autossh, or even init to allow easy restarting or treating it like a service.

12.1.2 Single use connection

If you just want to do a one time tunnel then the easiest thing to do is run the ssh with -f so that it detaches after the authentication phase and have it run a short sleep on the remote system. Then once it detaches connect to it with whatever insecure client you were tunneling for. Once both the sleep and the tunneled connection are closed the ssh session will silently exit. Here is an example for tunneling mysql: 'ssh -f -L 13306:mysql.domain.tld:3306 domain.tld sleep 60'. Then you can connect to mysql using 'mysql -hlocalhost -P13306'. If you stay in mysql for more than 60 seconds you will not notice the sleep exiting but the ssh session will end as soon as you close your TCP connection.

13 X-Windows tunneling

X-Windows tunneling is very similar to port forwarding except that X-Windows has an authentication system that OpenSSH can manipulate. In most cases simply initiating an ssh connection with either -X (or -Y depending on the X-Windows authentication configuration) is all that is needed. On the remote system your $DISPLAY will be set to something like localhost:10 which will be tunneled back to your client and then to whatever your $DISPLAY was set to when you ran ssh. This will allow you to run most X-Windows programs on a remote system and have them display on your local system with the traffic being encrypted (and probably compressed) by ssh.

14 Secure web proxy

Since web traffic is just TCP traffic obviously it can be tunneled just like any other TCP protocol. However, how many people only speak HTTP to a single remote system or would want to allocate a local port number for each remote system? If you want to tunnel all of your web browsing traffic through an ssh connection there is a better way. Instead of using the -L syntax use the -D localhost:localport syntax to create a socks proxy. If you run ssh with '-D localhost:8000 server' then you get a secure connection to server and a socks proxy running on localhost:8000. If you configure your web browser to use that socks proxy then all of your web traffic will proxy through the ssh connection to the remote server. Note again that the ssh portion of the connection is the only one that is secure. Once the http traffic leaves the ssh server it will be plain unencrypted http traffic. This is NOT an end-to-end encryption system like https. This is most commonly used to protect traffic as it goes through a public wi-fi but not much else.

15 Debugging

15.1 Client

To debug problems with the ssh client just run it with -v. If that doesn't help add another -v. Each -v you add will increase the verbosity. Thins to look for:

15.2 Server

Unfortunately many issues are on the server not the client and the server doesn't report things like why it refused a user's key authentication to the client. Therefore it is often necessary to get debugging information from the server. The easiest way to do this without disrupting the normal functions of sshd is to run an extra sshd on an alternate port number. If you were to run 'sshd -dDp 1022' you would get an extra sshd running in the foreground, outputting full debugging information in the terminal you ran it on, listening on tcp/1022, and it will exit after accepting a single ssh connection so there will not be extra information from other sessions to distract you. Things to look for are:

16 Performance enhancements

16.1 Compression

OpenSSH can compress the data stream before encrypting it. This can significantly increase performance over a slow connection. To use it either run ssh with -C for a one time use or set ``Compression on'' it in ~/.ssh/config.

16.2 Ciphers

OpenSSH supports multiple ciphers. Some are more secure than others while some are faster than others. If you are transferring a lot of data over ssh and one of the CPUs is struggling to keep up with the encryption then you can use '-c arcfour' or ``Cipher arcfour'' to use a weaker cipher that requires much less CPU resources. This is especially popular when doing backups on a private LAN or when using a NAS system with a weak embedded CPU.

16.3 High Performance eNabled SSH

HPN is a set of patches for OpenSSH that modify it to perform better on systems with multiple CPU cores. More information and the patches are available here: Some distros have packages with these patches already applied. Gentoo users have USE=hpn which is enabled by default.

About this document ...

OpenSSH Tips and Tricks for GOLUG

This document was generated using the LaTeX2HTML translator Version 2008 (1.71)

Copyright © 1993, 1994, 1995, 1996, Nikos Drakos, Computer Based Learning Unit, University of Leeds.
Copyright © 1997, 1998, 1999, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The command line arguments were:
latex2html -no_subdir -split 0 -show_section_numbers OpenSSH-2012.tex

The translation was initiated by Kevin Korb on 2012-01-04

next_inactive up previous
Kevin Korb 2012-01-04