mysql does not import apparmor profile correctly

I think I see this as well, simply doing an 'apt-get install mysql-server-5.6' on vivid leaves things in the following state after the installation completes:

  $ sudo aa-status
  [SNIP]
  2 processes are unconfined but have a profile defined.
     /usr/sbin/dnsmasq (665)
     /usr/sbin/mysqld (9186)

which suggests that something is going wrong in the rats nest of mysql.postinst/invoke-rc.d. Is it possible that somehow the sysv init script /etc/init.d/mysql is getting invoked instead of the upstart job? (... as that script does not load the mysql apparmor profile before starting mysql, unlike the upstart job).

This bug manifested in upgrade issues for users upgrading from mysql-5.5 (over the last day for any users following Vivid).

An update is on its way. Some comments on what happened:

The postinst (mysql-server-5.6.postinst) runs mysqld directly for bootstrapping before the "service" is started. On first install, this was unconfined, since AppArmor profile loading happens afterwards. On upgrade, this was on the previously loaded profile, not any new one shipped with the upgrade. I've fixed this with a workaround and have filed bug 1435368 to track a proper fix.

Also debian/rules was calling dh_apparmor at a point after dh_installinit, causing the ordering in the postinst to cause the new AppArmor profile to be loaded *after* the mysqld service was started. I've filed bug 1435452 because I think the dh sequencer should handle the ordering, but I've also uploaded the ordering fix manually.

The two bugs both meant that on upgrade the previous AppArmor profile was used, rather than the new one that permits mysqld access to the new /etc/mysql/mysql.conf.d/ directory. Previously, the upstart job manually loaded the profile before job start anyway. Now with systemd, we rely on dh_installinit and dh_apparmor to do the right thing entirely. This didn't work, so caused things to fail on upgrade.

Users would have seen a 10 minute delay while systemd timed out on starting the mysqld service unit, followed by a failure for mysqld to start anyway. Loading AppArmor profiles correctly would have worked around this.

Status changed to 'Confirmed' because the bug affects multiple users.

This bug was fixed in the package mysql-5.6 - 5.6.23-1~exp1~ubuntu4

---------------
mysql-5.6 (5.6.23-1~exp1~ubuntu4) vivid; urgency=medium

  * Ubuntu upload from Debian VCS commit ab01ca7.

mysql-5.6 (5.6.23-1~exp1) UNRELEASED; urgency=medium

  [ Robie Basak ]
  * New upstream release.

  [ Norvald H. Ryeng ]
  * Remove unused build options.
  * Add libedit dependency to mysql-client-core-5.6.
  * Remove chmods not needed after fix for upstream bug #68517.
  * Lintian fixes.

  [ Bjoern Boschman ]
  * Removed hardening-wrapper from build-deps.
  * Added python dep for mysql-testsuite-5.6.
  * Added dpkg-dev dep.
  * Cleanup a bit more.

  [ Robie Basak ]
  * Add dep8 smoke test.
  * Re-enable libmysqld-pic, libmysqld-dev, libmysqlclient18 and
    libmysqlclient-dev (renamed accordingly) to replace binary packages
    provided by src:mysql-5.5.
  * Add mysql-server, mysql-client and mysql-testsuite to be switched
    over from src:mysql-5.5.
  * Import mysql-common from src:mysql-5.5 for 5.5 -> 5.6 switchover.
  * Cherry-pick relevant parts of Ubuntu delta from mysql-5.5:
    - d/mysql-server-5.5.postrm: restoring postrm debhelper token to get
      proper behaviors from dh_installinit and dh_apparmor.
    - Add Apport hook.

  [ James Page ]
  * d/tests/upstream: Add main.ctype_uca to skip-test list as it requires
    writable /usr.
  * d/p/fix-mysqlhotcopy-test-failure.patch: Add return code 255 to the
    list of allowed return codes.
  * d/tests/control: Re-enable upstream DEP-8 tests.

  [ Otto Kekäläinen ]
  * Added parallel build support via DEB_BUILD_OPTIONS

  [ Otto Kekäläinen ]
  * Remove makeflags from cmake command and output MAKEFLAGS for debugging
    purposes just before make is run

  [ Akhil Mohan ]
  * Provisioned dep8 test case upstream separated to run as non-root

  [ Norvald H. Ryeng ]
  * Simplify and stabilize tests
  * Remove USE_STATIC_MYSQLD
  * Add test plan to README.Maintainer
  * Add --force rationale to test plan

  [ Akhil Mohan ]
  * Make builds parallel Use MAKEFLAGS for parallel builds

  [ Otto Kekäläinen ]
  * Move mysql_plugin to server package as it controls the server plugins
  * Wrap-and-sort
  * Remove legacy 5.0->5.1 transition code that refers a internal-use-only
    -directory that does not even exist anymore.
  * Removed all unnecessary .dirs as most of the directories are generated
    automatically
  * Don't emit extra output at init invocation Many users have complained that
    they're database is corrupt because they tried to start the database and it
    refuses to do so, and at the same time emitting a message with the word
    "corrupt". this is just plain wrong and confusing, so I'm removing it.

  [ Akhil Mohan ]
  * added systemd service profile and script

  [ Norvald H. Ryeng ]
  * Remove special build options for taocrypt

  [ Robie Basak ]
  * Drop unused variable definition
  * Create /etc/mysql manually in debian/rules
  * Replace conf.d/.keepme with an empty directory

  [ Akhil Mohan ]
  * Standards version moved up to 3.9.6 in d/control
  * Added Systemd service profile and script

  [ Robie Basak ]
  * wrap-and-sort for systemd change
  * Drop unnecessary binary target
  ...

here you can see my mysql-server version.
I have the correct version (5.6.23-1~exp1~ubuntu4) but mysql.service won't start.

ldisanto@luca-k53sd:~$ aptitude show mysql-server
Pacchetto: mysql-server
Stato: installato
Installato automaticamente: no
Versione: 5.6.23-1~exp1~ubuntu4
Priorità: opzionale
Sezione: database
Responsabile: Ubuntu Developers <email address hidden>
Architettura: all
Dimensione pacchetto installato: 118 k
Dipende: mysql-server-5.6
Fornito da: percona-server-server-5.6, percona-xtradb-cluster-server-5.6
Descrizione: MySQL database server (metapackage depending on the latest version)
 This is an empty package that depends on the current "best" version of mysql-server (currently mysql-server-5.6), as determined by the MySQL maintainers.
 Install this package if in doubt about which MySQL version you need. That will install the version recommended by the package maintainers.

 MySQL is a fast, stable and true multi-user, multi-threaded SQL database server. SQL (Structured Query Language) is the most popular database query language
 in the world. The main goals of MySQL are speed, robustness and ease of use.
Homepage: http://dev.mysql.com/

 sudo mkdir /var/run/mysqld
 sudo chmod 775 /run/mysqld/
 sudo chown root:mysql /run/mysqld/
made it work for me

@Luca

OK, then you are either affected by a different bug or have some kind of system configuration. Please file a separate bug to avoid cluttering up this one.

@Albert: it works but /run/mysqld wil be remove after a reboot. See https://bugs.launchpad.net/ubuntu/+source/mysql-5.6/+bug/1435823/comments/3

@Robie,

This has occurred since the package has landed, after a reboot the following occurs:

» systemctl status mysql.service
● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: activating (start-post) since Tue 2015-03-24 18:23:10 SAST; 7min ago
 Main PID: 16722 (code=exited, status=0/SUCCESS); : 16723 (mysql-systemd-s)
   CGroup: /system.slice/mysql.service
           └─control
             ├─16723 /bin/bash /usr/share/mysql/mysql-systemd-start post
             └─20513 sleep 1

Mar 24 18:23:10 host systemd[1]: mysql.service holdoff time over, scheduling restart.
Mar 24 18:23:10 host systemd[1]: Starting MySQL Community Server...
Mar 24 18:23:10 host mysqld_safe[16722]: 150324 18:23:10 mysqld_safe Can't log to error log and syslog at the same time. Remove all --log-error configuration options for --syslog to take effect.
Mar 24 18:23:10 host mysqld_safe[16722]: 150324 18:23:10 mysqld_safe Logging to '/var/log/mysql/error.log'.
Mar 24 18:23:10 host mysqld_safe[16722]: mkdir: cannot create directory ‘/var/run/mysqld’: Permission denied
Mar 24 18:23:10 host mysqld_safe[16722]: chown: cannot access ‘/var/run/mysqld’: No such file or directory
Mar 24 18:23:10 host mysqld_safe[16722]: chmod: cannot access ‘/var/run/mysqld’: No such file or directory
Mar 24 18:23:10 host mysqld_safe[16722]: 150324 18:23:10 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

Running the workaround in comment #6 seems start the service correctly, however it will have to be re-applied after a reboot as pointed out in comment #7.

● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2015-03-24 18:33:48 SAST; 2min 38s ago
  Process: 22041 ExecStartPost=/usr/share/mysql/mysql-systemd-start post (code=exited, status=0/SUCCESS)
  Process: 22038 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
 Main PID: 22040 (mysqld_safe)
   CGroup: /system.slice/mysql.service
           ├─22040 /bin/sh /usr/bin/mysqld_safe
           └─22401 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306

Mar 24 18:33:47 host systemd[1]: Starting MySQL Community Server...
Mar 24 18:33:47 host mysqld_safe[22040]: 150324 18:33:47 mysqld_safe Can't log to error log and syslog at the same time. Remove all --log-error configuration options for --syslog to take effect.
Mar 24 18:33:47 host mysqld_safe[22040]: 150324 18:33:47 mysqld_safe Logging to '/var/log/mysql/error.log'.
Mar 24 18:33:47 host mysqld_safe[22040]: 150324 18:33:47 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Mar 24 18:33:48 host systemd[1]: Started MySQL Community Server.

Users unfamiliar with problem, might try do a remove and install, but my oh my is that another story. If this should be filed as a seperate bug, let me know I'll be glad to do so.

Thanks

There were two separate bugs affecting the start of the mysqld service after the switch from 5.5 to 5.6 (and to systemd from upstart or System V init at the same time, which was after my QA work and why this breakage went undetected before - sorry!).

This bug is about the AppArmor policy load problem, which is now fixed. The other issue is about the creation of /var/run/mysqld, which is on my list of things to look at and is now tracked in bug 1435823. There is a workaround available in that bug.

Thanks Robie. I'll keep a eye out and also help out if need be for bug 1435823.