So the issue is that the current version of openssl/ssh require random to finish init before it will allow connections. This could also be causing problems with any openssl program.
Options:
1) Use rng-tools to allow use of the rpi hardware entropy generations. This program is has quite a few dependencies.
2) Use haveged. Please dont start a crypto battle here. haveged has no additional dependencies.
I've sent up the haveged extension.....once installed, just insert /usr/local/sbin/haveged at the beginning of /opt/bootlocal.sh
I'll update rng-tools, but there are more dependencies for me to chase first.