[Solved] ssh key authentication from remote terminal fails after Core update

[Leee]

I've installed Core 14.0 alongside Core 13.1 (both are x86_64).  The 14.0 install loads the same extensions (but from the 14.x repo, of course) as 13.1 loads and uses a copy of the mydata.tgz from 13.1.
The contents of    /home/tc/.ssh/    and    /usr/local/etc/ssh/    are included in the backup.

sshd is started from /opt/bootsync.sh and, on either version, it works and accepts logins.
ssh login with key authentication from localhost works as expected.
However, ssh login from a remote host (running MS Windows) using key authentication works from Core 13.1 but falls through to password authentication on Core 14.0.
Both versions are using the same host key  - it's stored in   /usr/local/etc/ssh/    and is in the backup and no "man-in-the-middle" warnings appear.  Both are using the same    authorized_keys    file.

Any ideas what I'm missing?

[Paul_123]

Check the sshd_config between the two versions.  typically: /usr/local/etc/ssh/sshd_config

More than likely the core 13.1 allowed for a less secure Authentication, and it is now blocked.

What is your windows client?

[Leee]

Hi Paul_123,
The sshd_config is also in the mydata.tgz brought over from 13.1.  To be thorough, I did just now verify the two copies of sshd_config with md5sum and they match.
The Windows client is openssh running under cygwin on Windows 7.
I just now tried opening a new terminal session in cygwin to see if there was just something in the session, but it still fails with key authentication and succeeds with password.
Just for kicks, I tried killing off ssh-agent on the client and restarting it - same results.  I didn't really expect anything new there.

Also specifying the key on the client command line ( with   -i   ) yields the same results.

[Rich]

Hi Leee
I ran a diff on  /usr/local/etc/ssh/sshd_config.orig  between TC13 and TC14:

Code: [Select]

rich@tcbox:~$ diff -u open13/usr/local/etc/ssh/sshd_config.orig open14/usr/local/etc/ssh/sshd_config.orig
--- open13/usr/local/etc/ssh/sshd_config.orig   2021-03-03 15:20:54.000000000 +0100
+++ open14/usr/local/etc/ssh/sshd_config.orig   2023-04-04 14:27:09.000000000 +0200
@@ -1,4 +1,4 @@
-#      $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $
+#      $OpenBSD: sshd_config,v 1.104 2021/07/02 05:11:21 dtucker Exp $
 
 # This is the sshd server system-wide configuration file.  See
 # sshd_config(5) for more information.
@@ -58,7 +58,7 @@
 #PermitEmptyPasswords no
 
 # Change to no to disable s/key passwords
-#ChallengeResponseAuthentication yes
+#KbdInteractiveAuthentication yes
 
 # Kerberos options
 #KerberosAuthentication no
@@ -72,13 +72,13 @@
 
 # Set this to 'yes' to enable PAM authentication, account processing,
 # and session processing. If this is enabled, PAM authentication will
-# be allowed through the ChallengeResponseAuthentication and
+# be allowed through the KbdInteractiveAuthentication and
 # PasswordAuthentication.  Depending on your PAM configuration,
-# PAM authentication via ChallengeResponseAuthentication may bypass
-# the setting of "PermitRootLogin without-password".
+# PAM authentication via KbdInteractiveAuthentication may bypass
+# the setting of "PermitRootLogin prohibit-password".
 # If you just want the PAM account and session checks to run without
 # PAM authentication, then enable this but set PasswordAuthentication
-# and ChallengeResponseAuthentication to 'no'.
+# and KbdInteractiveAuthentication to 'no'.
 #UsePAM no
 
 #AllowAgentForwarding yes
It appears  ChallengeResponseAuthentication
was replaced with  KbdInteractiveAuthentication.

I'm guessing TC14 is failing because  KbdInteractiveAuthentication
is not defined because you are restoring the TC13 version of
/usr/local/etc/ssh/sshd_config.

Adding the  KbdInteractiveAuthentication  variable to your TC13
backup of  /usr/local/etc/ssh/sshd_config  might allow it to work
for TC13 and TC14.

[GNUser]

Hi, Leee. TCL14 x86_64 uses openssh version 9.3p1. Since version 8.8, openssh server new default is to reject key authentication from clients using RSA + SHA-1 algorithm. This change has been causing problems for people because the rejected algorithm is commonly used by older ssh clients.

Try upgrading the ssh client on your remote (MS Windows) machine. If that's not possible, you can configure your TCL14 ssh server to continue accepting the old algorithm by adding this to your ssh server configuration file:

Code: [Select]

HostkeyAlgorithms +ssh-rsa
PubkeyAcceptedAlgorithms +ssh-rsa

[Paul_123]

Typically there is not much need to adjust the default sshd config that comes with core.  I would just try to remove it from your backup, or at least make your edits in the new config file and then back it up.

I'm not familiar with cygwin, but I think it uses openssh too, what version is it using?  More than likely the ssh client is trying to authenticate using a newly restricted key
 exchange algorithm.  Run your ssh client connection from cygwin with some verbosity (-v or -vv) and see what it says.

[Paul_123]

Reference release notes.  Looks like your client needs to be 5.7 or newer(if it is openssh)


https://www.openssh.com/txt/release-8.2


https://www.openssh.com/txt/release-8.8


Note, that this is not an issue with the keys themselves, but with how ssh signs them when negotiating.

[Leee]

Still no luck on this.
The ssh version on the client (under cygwin) is OpenSSH_6.6p1, so it's newer than 5.7.
I've checked logging in from another client (Core 12.0) and key authentication works from there, so I'm still supposing it has to do with the client version on cygwin.
I've got a bunch of other irons in the fire right now and want his to "just work", so I've simply reverted Openssh in Core 14 to the Openssh from Core 13.x.  Not a workaround that I would usually approve of but it works.  I'll post here when I finally solve the issue (unless I retire the Win7/Cygwin host in the mean time).

[GNUser]

Still no luck on this.

Hi, Leee. Sorry to hear. Did you try the tweak to sshd_config that I suggested in my previous post?

[Leee]

I did try that, and it still didn't allow key-auth login from the Windows host.

I'm dead in the water at the moment because, in an unrelated change, I managed to wreck the boot device (which is just a modest-sized USB stick, so no biggie in the grand scheme of things) such that the grub menu appears but -none- of the several Linux installations boots.  So I'll be migrating the whole mess to a newer (and much larger and faster USB3.0) USB stick before thoroughly re-testing/troubleshooting the SSH thing.  But not today as I'm otherwise occupied for the next ten hours or so.

[GNUser]

I did try that, and it still didn't allow key-auth login from the Windows host.

Darn. Your problem sure sounds like client using an algorithm that the new server version's default is to reject. If this is indeed the problem, the client must be using an algorithm other than RSA+SHA1.

I'm sorry to hear that your boot device got borked. Hopefully you're up and running again soon. At least with TCL it's easy to rebuild the OS from scratch since that's exactly what happens at every boot (core.gz + onboot.lst + mydata.tgz)

[aus9]

Hi

the grub menu appears but -none- of the several Linux installations boots

and I know you are going to make a bigger usb stick thingee....but it does not take long to attempt to re-embed grub with its booting files.

Become root and keep root terminal if this interests you
run

Code: [Select]

sudo su
blkidinsert your usb stick and repeat blkid ....let pretend new /dev/sdc* appears assumes one internal drive, one separate boot device and target usb device.

your boot dir may be on the root of your usb or a separate partition lets pretend its on the first partition below ( sdc1)

now mount target device somewhere easy to remember with example

Code: [Select]

mkdir /tmp/z
mount /dev/sdc1 /tmp/z
find /tmp/z -name grub.cfg # should give you a hit otherwise we can not continue...note the pathway may vary comments below
grub-install --boot-directory=/tmp/z/boot /dev/sdc
tmp/z/boot may not work depending on your hit.  grub does not need a boot dir it works with a grub dir just as well
Your hit might read/tmp/z/grub/grub.cfg.....so above would change to /tmp/z/grub

good luck

You might need to make extra dirs if grub.cfg is not on sdc1 (eg) and mount each until the find command finds it

[Leee]

Alright then... back at this.
I installed the openssh.tcz from the 14.x repo and copied the provided sshd_config.orig to sshd_config.  There's nothing special about the sshd_config I was using in 13.1 except that I had allowed X11Forwarding, 

Following up on Rich's suggestion in reply #5 -

• Verified "ChallengeResponseAuthentication" is not present
• Verified "KbdInteractiveAuthentication" is present and set to "yes" (the default value).
• Tested: does not allow key authenticated login from Windows/cygwin client.

I set the -v  (verbose) option on the command line (client side) and got the following:

Code: [Select]

leee@WinPC ~$ ssh -vY tc@192.168.1.95
OpenSSH_6.6, OpenSSL 1.0.1g 7 Apr 2014
debug1: Connecting to 192.168.1.95 [192.168.1.95] port 22.
debug1: Connection established.
debug1: identity file /.ssh/id_rsa type -1
debug1: identity file /.ssh/id_rsa-cert type -1
debug1: identity file /.ssh/id_dsa type -1
debug1: identity file /.ssh/id_dsa-cert type -1
debug1: identity file /.ssh/id_ecdsa type -1
debug1: identity file /.ssh/id_ecdsa-cert type -1
debug1: identity file /.ssh/id_ed25519 type -1
debug1: identity file /.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.3
debug1: match: OpenSSH_9.3 pat OpenSSH* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-sha1-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-sha1-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA cf:bd:6c:f9:7d:9b:8b:b3:ed:76:c5:e1:25:28:04:b0
debug1: Host '192.168.1.95' is known and matches the ECDSA host key.
debug1: Found key in /.ssh/known_hosts:12
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/leee/.ssh/id_rsa
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Trying private key: /.ssh/id_rsa
debug1: Trying private key: /.ssh/id_dsa
debug1: Trying private key: /.ssh/id_ecdsa
debug1: Trying private key: /.ssh/id_ed25519
debug1: Next authentication method: keyboard-interactive
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: password
tc@192.168.1.95's password:

Tried starting sshd in debug mode, then logging in as before.  The server output was:

Code: [Select]

Server listening on 0.0.0.0 port 22.
userauth_pubkey: signature algorithm ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Accepted password for tc from 192.168.1.101 port 59065 ssh2

So it looks like I'll have to try updating the client side openssh - but I seem to vaguely recall 32-bit Windows / cygwin being an issue.
---
@aus9 - I did copy everything from the borked USB stick to another, larger USB stick that was already set up with grub2-multi ( per juanito's instructions from 2015-12-23 https://forum.tinycorelinux.net/index.php/topic,19364.0.html ) and it's all booting as expected. 

Since the old USB stick was set up for BIOS systems only (not uefi), I thought I'd rebuild it from scratch to allow either and keep it around as a backup.  Just curious though... In your post regarding fixing up the broken bootloader without doing a complete reinstall (for which I thank you and which I'll probably try as an exercise in non-destructive repair/rescue), you mentioned creating a new mountpoint for the as /tmp/z.  Why not just let it use the default mountpoint, which would be /mnt/sdc1 in the example?

[curaga]

Your server log says you should follow GNUser's post?

[patrikg]

One neet function with sshd is you can execute this line to get the running config:

Code: (bash) [Select]

sudo sshd -T