$ complete -W "$(echo $(grep '^ssh ' .bash_history | sort -u | sed 's/^ssh //'))" ssh
$ ssh [TAB]
See the following article for more details:
http://www.commandlinefu.com/commands/view/2759/ssh-autocomplete
Update: I just realized this morning that there's one more thing you need to do to get this working properly. Somehow you need to source your custom complete commands when you open a new terminal. I use bash, so I vi'd my .bashrc file and added my custom completion commands. There's also a directory called /etc/bash_completion.d for these scripts, but I felt a little squeamish putting my custom ones there.
# enable programmable completion features (you don't need to
# enable this, if it's already enabled in /etc/bash.bashrc
# and /etc/profile sources /etc/bash.bashrc).
if [ -f /etc/bash_completion ] && ! shopt -oq posix; then
. /etc/bash_completion
# custom completions
complete -W "$(echo $(grep '^ssh ' .bash_history | sort -u | sed 's/^ssh //'))" ssh
fi