Disappearing /var/run/mysqld causes mysqld to fail after reboot

Status changed to 'Confirmed' because the bug affects multiple users.

Create a newfile /etc/tmpfiles.d/mysql.conf:

# systemd tmpfile settings for mysql
# See tmpfiles.d(5) for details

d /var/run/mysqld 0755 mysql mysql -

After reboot, mysql should start normally.

@Luca

Does Bruno's fix work for you? I'd like to know if so, because that suggests that we have a problem with /var/run/mysqld not recreated after boot which is certainly a bug. I need to figure out steps to reproduce in order to fix it.

Hi,
With Bruno's fix works

Thank you!!

@Luca: prego;)

I guess that /run/mysqld is not created after boot because in /lib/systemd/mysql.service, mysqld_safe script (which seems to be responsible for its creation) is invoked with user/group mysql/mysql and /run is only writable by root.

Comment #3 works for me as well.

Comment #3 fixed it for me, too.

Comment #3 fixed it for me, too.

Comment #3 works perfect, thank you!

This bug was fixed in the package mysql-5.6 - 5.6.23-1~exp1~ubuntu5

---------------
mysql-5.6 (5.6.23-1~exp1~ubuntu5) vivid; urgency=medium

  * Ubuntu upload from Debian VCS commit 804459b.

mysql-5.6 (5.6.23-1~exp1) UNRELEASED; urgency=medium

  [ Robie Basak ]
  * New upstream release.

  [ Norvald H. Ryeng ]
  * Remove unused build options.
  * Add libedit dependency to mysql-client-core-5.6.
  * Remove chmods not needed after fix for upstream bug #68517.
  * Lintian fixes.

  [ Bjoern Boschman ]
  * Removed hardening-wrapper from build-deps.
  * Added python dep for mysql-testsuite-5.6.
  * Added dpkg-dev dep.
  * Cleanup a bit more.

  [ Robie Basak ]
  * Add dep8 smoke test.
  * Re-enable libmysqld-pic, libmysqld-dev, libmysqlclient18 and
    libmysqlclient-dev (renamed accordingly) to replace binary packages
    provided by src:mysql-5.5.
  * Add mysql-server, mysql-client and mysql-testsuite to be switched
    over from src:mysql-5.5.
  * Import mysql-common from src:mysql-5.5 for 5.5 -> 5.6 switchover.
  * Cherry-pick relevant parts of Ubuntu delta from mysql-5.5:
    - d/mysql-server-5.5.postrm: restoring postrm debhelper token to get
      proper behaviors from dh_installinit and dh_apparmor.
    - Add Apport hook.

  [ James Page ]
  * d/tests/upstream: Add main.ctype_uca to skip-test list as it requires
    writable /usr.
  * d/p/fix-mysqlhotcopy-test-failure.patch: Add return code 255 to the
    list of allowed return codes.
  * d/tests/control: Re-enable upstream DEP-8 tests.

  [ Otto Kekäläinen ]
  * Added parallel build support via DEB_BUILD_OPTIONS

  [ Otto Kekäläinen ]
  * Remove makeflags from cmake command and output MAKEFLAGS for debugging
    purposes just before make is run

  [ Akhil Mohan ]
  * Provisioned dep8 test case upstream separated to run as non-root

  [ Norvald H. Ryeng ]
  * Simplify and stabilize tests
  * Remove USE_STATIC_MYSQLD
  * Add test plan to README.Maintainer
  * Add --force rationale to test plan

  [ Akhil Mohan ]
  * Make builds parallel Use MAKEFLAGS for parallel builds

  [ Otto Kekäläinen ]
  * Move mysql_plugin to server package as it controls the server plugins
  * Wrap-and-sort
  * Remove legacy 5.0->5.1 transition code that refers a internal-use-only
    -directory that does not even exist anymore.
  * Removed all unnecessary .dirs as most of the directories are generated
    automatically
  * Don't emit extra output at init invocation Many users have complained that
    they're database is corrupt because they tried to start the database and it
    refuses to do so, and at the same time emitting a message with the word
    "corrupt". this is just plain wrong and confusing, so I'm removing it.

  [ Akhil Mohan ]
  * added systemd service profile and script

  [ Norvald H. Ryeng ]
  * Remove special build options for taocrypt

  [ Robie Basak ]
  * Drop unused variable definition
  * Create /etc/mysql manually in debian/rules
  * Replace conf.d/.keepme with an empty directory

  [ Akhil Mohan ]
  * Standards version moved up to 3.9.6 in d/control
  * Added Systemd service profile and script

  [ Robie Basak ]
  * wrap-and-sort for systemd change
  * Drop unnecessary binary target
  ...

Hi,

I am not an expert,so I do'nt know how to apply the Bruno's fix

When I start mysql I get:

Sep 11 11:08:52 serv mysql[25802]: Checking for tables which need an up...e
Sep 11 11:08:52 serv mysql[25802]: not closed cleanly..
Hint: Some lines were ellipsized, use -l to show in full.

I understand I have to

Create a newfile /etc/tmpfiles.d/mysql.conf:

What does it mean " d /var/run/mysqld 0755 mysql mysql - " ? Delete?

Thank You to all mysql experts that can solve this question.

Yours Sincerely,
Alex

Hi al,

As per tempfile(1) (man tempfile) that line creates a new directory at /var/run/mysqld with 0755 permissions and sets the owner and group to mysql.

@kiaikim

Why did you add a task for mysql-5.7? This bug was fixed in 5.6.25-2 which predates mysql-5.7.

@racb
I'm sorry. I think I got webpages confused. I had the same problem with 5.7.17 on Ubuntu 16.04. I looked all over and tried whatever I could find. Must have thought I tried #3, but I did chown of /var/run/mysqld/ with mysql service running, which also worked. If you'd like I can start over, see if the problem persists and try #3. But probably better to see if others have this problem.

Actually, I created /var/run/mysqld/, and then did chown.

No problem. Feel free to file a separate bug against mysql-5.7 and link to it from here. If you can, please find steps that a developer can use to reproduce the problem. If you can't, then feel free to file a bug but please set the status to Incomplete. Others affected can then find and communicate in the bug with you to try and figure out steps to reproduce, or a technical explanation of what is happening.

Note though that the community doesn't generally follow bugs, so to find others to get help you are more likely to get help in other places. See http://www.ubuntu.com/support/community for community support options.

This issue seems to be relevant also in the most recent update from apt for Ubuntu 14.04. (Note i am running percona version) I just ran the update this morning and am now seeing this same issue. I've tried to implement the tmfiles.d solution but that does not seem to take effect. I _can_ manually create the /var/run/mysqld/ folder and chown same as kiakim, but this manual step is a bit wonky.

When will the 'fix' be available? Or, is this now an issue for the percona team to address?

This may or may not be relevant... but i did notice in the release notes for the 5.6.35 version of mysql (the version now being used by our ubuntu 14.04 update)... it looks like work has been done on the mysqld_safe script and use of chown is removed. Is it possible the necessary security lockdown re-introduced the bug.

https://dev.mysql.com/doc/relnotes/mysql/5.6/en/news-5-6-35.html

Same issue here, workaround by tmpfiles.d.

> When will the 'fix' be available?

This bug has already been fixed. If the issue has somehow recurred, or occurred in a different package, then please file a new bug with a full report (packages and versions used, steps to reproduce, etc) and link to this one. If you do not do this, it will not appear in our reporting and thus will never get fixed.

I'm not the only one having this problem with 5.7. See the first comment under http://stackoverflow.com/a/42640731/2668783

> I'm not the only one having this problem with 5.7.

That's fine, but as I said, please file a separate bug with a full report.

This bug is appear AGAIN !!!

# systemd-tmpfiles --create
# ls -d /var/run/mysqld
/var/run/mysqld

# ls -dF /var/run/mysqld
/var/run/mysqld/

# service mysql start
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.

# ls -dF /var/run/mysqld
ls: cannot access '/var/run/mysqld': No such file or directory

# systemctl status mysql.service
● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2020-04-08 13:20:58 CST; 1min 10s ago
  Process: 15705 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=1/FAILURE)

Apr 08 13:20:58 localhost systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Apr 08 13:20:58 localhost systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
Apr 08 13:20:58 localhost systemd[1]: Stopped MySQL Community Server.
Apr 08 13:20:58 localhost systemd[1]: mysql.service: Start request repeated too quickly.
Apr 08 13:20:58 localhost systemd[1]: mysql.service: Failed with result 'exit-code'.
Apr 08 13:20:58 localhost systemd[1]: Failed to start MySQL Community Server.

# mysqld --version
mysqld Ver 5.7.29-0ubuntu0.18.04.1 for Linux on x86_64 ((Ubuntu))

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic

Hello gene,

I doubt the problem you are observing is related to the issue described in this report. In this bug description the failure mode is:

mar 24 08:41:00 luca-k53sd mysqld_safe[951]: mkdir: impossibile creare la directory "/var/run/mysqld": Permesso negato
mar 24 08:41:00 luca-k53sd mysqld_safe[951]: chown: impossibile accedere a "/var/run/mysqld": File o directory non esistente

where the errors are "Permission denied" and "No such file or directory". This doesn't happen in your case, what happens is that systemd can't start the service and cleans up the tmp directory after giving up.

Anyway if you are still convinced you are hitting this problem again please file a separate report for mysql 5.7, as this one is specifically for version 5.6. See Robie's comments above. Thanks!