sshd failed on config reload

Bug #1771340 reported by Tronde
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
openssh (Debian)
Fix Released
Unknown
openssh (Ubuntu)
Fix Released
Low
Unassigned
Xenial
Fix Released
Low
Andreas Hasenack

Bug Description

[Impact]

sshd doesn't check the configuration when reloading.

If a user generates an invalid configuration file, sshd will shut down and not come back up when the user issues a reload.

[Test Case]

$ lxc launch ubuntu:xenial tester
$ lxc exec tester bash

# echo "blah blah" >>/etc/ssh/sshd_config
# systemctl reload sshd
Job for ssh.service failed because the control process exited with error code. See "systemctl status ssh.service" and "journalctl -xe" for details.
# systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2018-08-21 18:15:41 UTC; 19s ago

* The service should have checked the config file, failed to reload, but remained active in its current configuration. In this case ssh has shut down.

[Regression Potential]

This code will only trigger on an invalid configuration file (in which case sshd would not load anyway), so there should be no regressions.

[Other Info]

autopkgtest [13:45:46]: test regress: -----------------------]
autopkgtest [13:45:47]: test regress: - - - - - - - - - - results - - - - - - - - - -
regress PASS
autopkgtest [13:45:47]: @@@@@@@@@@@@@@@@@@@@ summary
regress PASS

[Original Description]

After adding some lines to /etc/ssh/sshd_config I tried to reload the configuration with the command:

```
sudo systemctl reload sshd
```

No error message was returned. So I assumed that the sshd was running with the current config. But `sudo systemctl status sshd` told me that the service failed due to a wrong option in /etc/ssh/sshd_config. Please see the following output:

~~~
:~$ sudo vim /etc/ssh/sshd_config
:~$ sudo systemctl reload sshd
:~$ sudo systemctl status sshd
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Di 2018-05-15 10:00:04 CEST; 8s ago
  Process: 12089 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
  Process: 7536 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255)
 Main PID: 7536 (code=exited, status=255)
~~~

I would expect that a warning or error message is returned when the service fails while reloading it's configuration.

A fix for this behaviour would be appreciated.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: openssh-server 1:7.2p2-4ubuntu2.4
ProcVersionSignature: Ubuntu 3.13.0-112.159-generic 3.13.11-ckt39
Uname: Linux 3.13.0-112-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.17
Architecture: amd64
Date: Tue May 15 10:18:25 2018
InstallationDate: Installed on 2013-01-10 (1950 days ago)
InstallationMedia: Ubuntu-Server 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120817.3)
SourcePackage: openssh
UpgradeStatus: Upgraded to xenial on 2017-03-12 (428 days ago)
mtime.conffile..etc.pam.d.sshd: 2017-03-13T19:59:01.965420

Related branches

Revision history for this message
Tronde (tronde) wrote :
Revision history for this message
Tronde (tronde) wrote :

This bug seems to be fixed upstream in a newer version: [Debian Bug report logs - #865770
openssh-server fails to validate configuration before reloading, under systemd](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=865770)

Maybe it is possible to get a version update in Xenial?!

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Confirmed fixed in >= artful. We have two ExecReload items under [Service]:
[Service]
...
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID

tags: added: bitesize
Changed in openssh (Ubuntu):
status: New → Triaged
importance: Undecided → Low
Revision history for this message
Marcos Paulo de Souza (marcospsouza) wrote :

I can confirm this is working in Xenial. After change the sshd_config config and execute a systemctl restart it worked. The ssh.service file have only one ExecReload clause:

...
[Service]
EnvironmentFile=-/etc/default/ssh
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/bin/kill -HUP $MAINPID
...

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

The issue is reload, not restart.

You should:
- get a working normal sshd_config
- issue reload, confirm it works
- add an invalid option to sshd_config
- issue reload

The broken system will kill sshd, whereas the fixed one will refuse to reload but sshd will still be running.

Changed in openssh (Ubuntu):
assignee: nobody → Karl (kstenerud)
description: updated
description: updated
description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

marking as fix released in the devel task, since the fix is in cosmic.

Changed in openssh (Ubuntu):
status: Triaged → Fix Released
assignee: Karl (kstenerud) → nobody
Changed in openssh (Ubuntu):
status: Fix Released → In Progress
status: In Progress → Fix Released
Changed in openssh (Ubuntu Xenial):
status: New → In Progress
assignee: nobody → Karl (kstenerud)
Changed in openssh (Ubuntu Xenial):
importance: Undecided → Low
Revision history for this message
Brian Murray (brian-murray) wrote :

Could you provide links indicating that this is fixed in both Ubuntu 18.10 and Ubuntu 18.04? Thanks in advance.

Revision history for this message
Karl Stenerud (kstenerud) wrote :
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Tronde, or anyone else affected,

Accepted openssh into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:7.2p2-4ubuntu2.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in openssh (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-xenial
Changed in openssh (Debian):
status: Unknown → Fix Released
Revision history for this message
Tronde (tronde) wrote :

Hi there,

I tested version 1:7.2p2-4ubuntu2.5 from proposed but the issue still exists. Behavior is exactly as before.

Regards,
Tronde

tags: added: verification-failed-xenial
removed: verification-needed-xenial
Changed in openssh (Ubuntu Xenial):
assignee: Karl Stenerud (kstenerud) → Andreas Hasenack (ahasenack)
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

@tronde, I just tried and the fix worked for me.

With the proposed package:
root@xenial-ssh-reload:~# ps fxaw
  PID TTY STAT TIME COMMAND
    1 ? Ss 0:02 /sbin/init
   55 ? Ss 0:00 /lib/systemd/systemd-journald
...
 2443 ? Ss 0:00 /usr/sbin/sshd -D

Note the sshd pid: 2443

Reload fails after the config file is corrupted, as expected:
root@xenial-ssh-reload:~# echo "blah blah" >>/etc/ssh/sshd_config
root@xenial-ssh-reload:~# systemctl reload ssh
Job for ssh.service failed because the control process exited with error code. See "systemctl status ssh.service" and "journalctl -xe" for details.

But service is still running as before, same pid:
root@xenial-ssh-reload:~# ps fxaw
  PID TTY STAT TIME COMMAND
    1 ? Ss 0:02 /sbin/init
...
 2443 ? Ss 0:00 /usr/sbin/sshd -D

And status agrees:
root@xenial-ssh-reload:~# systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: active (running) (Result: exit-code) since Wed 2018-10-10 18:00:30 UTC; 1min 55s ago
  Process: 2491 ExecReload=/usr/sbin/sshd -t (code=exited, status=255)
  Process: 2442 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
 Main PID: 2443 (sshd)
    Tasks: 1
   Memory: 1.5M
      CPU: 24ms
   CGroup: /system.slice/ssh.service
           └─2443 /usr/sbin/sshd -D

Oct 10 18:00:30 xenial-ssh-reload systemd[1]: Starting OpenBSD Secure Shell server...
Oct 10 18:00:30 xenial-ssh-reload sshd[2443]: Server listening on 0.0.0.0 port 22.
Oct 10 18:00:30 xenial-ssh-reload sshd[2443]: Server listening on :: port 22.
Oct 10 18:00:30 xenial-ssh-reload systemd[1]: Started OpenBSD Secure Shell server.
Oct 10 18:01:01 xenial-ssh-reload systemd[1]: Reloading OpenBSD Secure Shell server.
Oct 10 18:01:01 xenial-ssh-reload sshd[2491]: /etc/ssh/sshd_config: line 89: Bad configuration option: blah
Oct 10 18:01:01 xenial-ssh-reload sshd[2491]: /etc/ssh/sshd_config: terminating, 1 bad configuration options
Oct 10 18:01:01 xenial-ssh-reload systemd[1]: ssh.service: Control process exited, code=exited status=255
Oct 10 18:01:01 xenial-ssh-reload systemd[1]: Reload failed for OpenBSD Secure Shell server.

Note how it logged that there was a bad config option (as a result of calling sshd -t before the actual reload).

Could you please double check? For me, this update is fine.

Revision history for this message
Tronde (tronde) wrote :

@ahasenack, of course I could double check. I've done so a few minutes ago and you are right. After trying to reload with a corrupted config file the reload failed but the service is still up and running.

Please see the following output for confirmation:

~~~
root@vbox-xenial:~# systemctl status sshd
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: active (running) since Do 2018-10-11 11:13:35 CEST; 2min 19s ago
 Main PID: 8917 (sshd)
   CGroup: /system.slice/ssh.service
           └─8917 /usr/sbin/sshd -D

Okt 11 11:13:35 vbox-xenial systemd[1]: Starting OpenBSD Secure Shell server...
Okt 11 11:13:35 vbox-xenial sshd[8917]: Server listening on 0.0.0.0 port 22.
Okt 11 11:13:35 vbox-xenial sshd[8917]: Server listening on :: port 22.
Okt 11 11:13:35 vbox-xenial systemd[1]: Started OpenBSD Secure Shell server.
root@vbox-xenial:~# echo "blah blah" >>/etc/ssh/sshd_config
root@vbox-xenial:~# systemctl reload sshd
Job for ssh.service failed because the control process exited with error code. See "systemctl status ssh.service" and "journalctl -xe" for details.
root@vbox-xenial:~# systemctl status sshd
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: active (running) (Result: exit-code) since Do 2018-10-11 11:13:35 CEST; 2min 51s ago
  Process: 9033 ExecReload=/usr/sbin/sshd -t (code=exited, status=255)
 Main PID: 8917 (sshd)
   CGroup: /system.slice/ssh.service
           └─8917 /usr/sbin/sshd -D

Okt 11 11:13:35 vbox-xenial systemd[1]: Starting OpenBSD Secure Shell server...
Okt 11 11:13:35 vbox-xenial sshd[8917]: Server listening on 0.0.0.0 port 22.
Okt 11 11:13:35 vbox-xenial sshd[8917]: Server listening on :: port 22.
Okt 11 11:13:35 vbox-xenial systemd[1]: Started OpenBSD Secure Shell server.
Okt 11 11:16:15 vbox-xenial systemd[1]: Reloading OpenBSD Secure Shell server.
Okt 11 11:16:15 vbox-xenial sshd[9033]: /etc/ssh/sshd_config: line 89: Bad configuration option: blah
Okt 11 11:16:15 vbox-xenial sshd[9033]: /etc/ssh/sshd_config: terminating, 1 bad configuration options
Okt 11 11:16:15 vbox-xenial systemd[1]: ssh.service: Control process exited, code=exited status=255
Okt 11 11:16:15 vbox-xenial systemd[1]: Reload failed for OpenBSD Secure Shell server.
root@vbox-xenial:~#
~~~

Sorry, that I didn't get it in the first try.

The update looks fine for my, too.

tags: added: verification-done-xenial
removed: verification-failed-xenial verification-needed
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Thanks for the confirmation @tronde, much appreciated.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssh - 1:7.2p2-4ubuntu2.5

---------------
openssh (1:7.2p2-4ubuntu2.5) xenial; urgency=medium

  * debian/systemd/ssh.service: Test configuration before starting or
    reloading sshd (LP: #1771340)

 -- Karl Stenerud <email address hidden> Tue, 21 Aug 2018 10:45:26 -0700

Changed in openssh (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Robie Basak (racb) wrote : Update Released

The verification of the Stable Release Update for openssh has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.