mySQL not starting since kernel update

16 posts / 0 new
Last post
#1 Sun, 09/22/2013 - 13:31
singh90

mySQL not starting since kernel update

Hi guys,

I posted on Experts-Exchange but havent had the fix that i require yet.

So I've been an idiot and upgraded the kernel on my virtualmin server... everything works smoothly except for mysql :|

no matter what i try, i am unable to bring the service back to life.

sudo service mysql start start: Job failed to start

uname -a Linux hostname-removed 3.2.0-53-generic #81-Ubuntu SMP Thu Aug 22 21:01:03 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

I am running Ubuntu 12.04 and Virtualmin 4.02 - I have several sites set up which depend on mysql. Each site worked perfectly until the upgrade!

Any ideas?

Sun, 09/22/2013 - 14:49
Locutus

Firstly, you're certainly not an idiot for updating the kernel. :) It's suggested to do so from time to time.

Second, if all else fails, you can always revert back to the previous kernel. You even get a grub boot menu where you can select it, since the automatic process keeps all the previous kernels.

Then, to know why MySQL fails to start you might want to check /var/log/syslog for error messages. Without further information about that, it's hard to say what could be the reason. If there's nothing in the syslog, start mysqld from the command line and check its output.

Sun, 09/22/2013 - 15:49
singh90

unfortunately to add complication this is a virtually hosted system so i cant access grub. unless this can be done from cmd line or VirtualMin CP?

and even further complication, im a bit of a noob :)

I have tried to start mysqld but it fails to start - the service is in a stop/waiting status and cant be stopped manually either.

Sun, 09/22/2013 - 21:08
Locutus

Please give us more details about "does not start", what exactly did you try and what messages do you get? What's in the syslog?

To select kernel during boot, you need console access. Does your virtual host provider offer that? Otherwise you can uninstall the new kernel's package using aptitude. But to be honest, I have doubts that the new kernel is the problem. New kernels rarely have such specific problems that one service doesn't work anymore but the rest does.

Mon, 09/23/2013 - 08:24
singh90

unfortunately I don't have console access :(

This is what I have tried so far.

/usr/bin/mysqld_safe --mysqld
130911 22:09:16 mysqld_safe Can't log to error log and syslog at the same time.  Remove all --log-error configuration options for --syslog to take effect.
130911 22:09:16 mysqld_safe Logging to '/var/log/mysql/error.log'.
130911 22:09:16 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130911 22:09:16 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

error.log

130911 20:39:11 [Note] /usr/sbin/mysqld: Shutdown complete

130911 22:09:16 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130911 22:09:16 [Note] Plugin 'FEDERATED' is disabled.
130911 22:09:16 InnoDB: The InnoDB memory heap is disabled
130911 22:09:16 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130911 22:09:16 InnoDB: Compressed tables use zlib 1.2.3.4
/usr/sbin/mysqld: Can't create/write to file '/tmp/ibzUySNF' (Errcode: 13)
130911 22:09:16  InnoDB: Error: unable to create temporary file; errno: 13
130911 22:09:16 [ERROR] Plugin 'InnoDB' init function returned error.
130911 22:09:16 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
130911 22:09:16 [ERROR] /usr/sbin/mysqld: unknown option '--mysqld'
130911 22:09:16 [ERROR] Aborting

130911 22:09:16 [Note] /usr/sbin/mysqld: Shutdown complete

130911 22:09:16 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

my.cnf

#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size       = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover         = BACKUP
#max_connections        = 100
#table_cache            = 64
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size        = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file        = /var/log/mysql/mysql.log
#general_log             = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size         = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem



[mysqldump]
quick
quote-names
max_allowed_packet = 16M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer = 16M

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

MySQL

#!/bin/sh -e
# upstart-job
#
# Symlink target for initscripts that have been converted to Upstart.

set -e

INITSCRIPT="$(basename "$0")"
JOB="${INITSCRIPT%.sh}"

if [ "$JOB" = "upstart-job" ]; then
    if [ -z "$1" ]; then
        echo "Usage: upstart-job JOB COMMAND" 1>&2
exit 1
    fi

    JOB="$1"
    INITSCRIPT="$1"
    shift
else
    if [ -z "$1" ]; then
        echo "Usage: $0 COMMAND" 1>&2
exit 1
    fi
fi

COMMAND="$1"
shift


if [ -z "$DPKG_MAINTSCRIPT_PACKAGE" ]; then
ECHO=echo
else
ECHO=:
fi

$ECHO "Rather than invoking init scripts through /etc/init.d, use the service(8)"
$ECHO "utility, e.g. service $INITSCRIPT $COMMAND"

# Only check if jobs are disabled if the currently _running_ version of
# Upstart (which may be older than the latest _installed_ version)
# supports such a query.
#
# This check is necessary to handle the scenario when upgrading from a
# release without the 'show-config' command (introduced in
# Upstart for Ubuntu version 0.9.7) since without this check, all
# installed packages with associated Upstart jobs would be considered
# disabled.
#
# Once Upstart can maintain state on re-exec, this change can be
# dropped (since the currently running version of Upstart will always
# match the latest installed version).

UPSTART_VERSION_RUNNING=$(initctl version|awk '{print $3}'|tr -d ')')

if dpkg --compare-versions "$UPSTART_VERSION_RUNNING" ge 0.9.7
then
    initctl show-config -e "$JOB"|grep -q '^  start on' || DISABLED=1
fi

case $COMMAND in
status)
    $ECHO
    $ECHO "Since the script you are attempting to invoke has been converted to an"
    $ECHO "Upstart job, you may also use the $COMMAND(8) utility, e.g. $COMMAND $JOB"
    $COMMAND "$JOB"
    ;;
start|stop)
    $ECHO
    $ECHO "Since the script you are attempting to invoke has been converted to an"
    $ECHO "Upstart job, you may also use the $COMMAND(8) utility, e.g. $COMMAND $JOB"
    if status "$JOB" 2>/dev/null | grep -q ' start/'; then
        RUNNING=1
    fi
    if [ -z "$RUNNING" ] && [ "$COMMAND" = "stop" ]; then
        exit 0
    elif [ -n "$RUNNING" ] && [ "$COMMAND" = "start" ]; then
        exit 0
    elif [ -n "$DISABLED" ] && [ "$COMMAND" = "start" ]; then
        exit 0
    fi
    $COMMAND "$JOB"
    ;;
restart)
    $ECHO
    $ECHO "Since the script you are attempting to invoke has been converted to an"
    $ECHO "Upstart job, you may also use the stop(8) and then start(8) utilities,"
    $ECHO "e.g. stop $JOB ; start $JOB. The restart(8) utility is also available."
    if status "$JOB" 2>/dev/null | grep -q ' start/'; then
        RUNNING=1
    fi
    if [ -n "$RUNNING" ] ; then
        stop "$JOB"
    fi
    # If the job is disabled and is not currently running, the job is
    # not restarted. However, if the job is disabled but has been forced into the
    # running state, we *do* stop and restart it since this is expected behaviour
    # for the admin who forced the start.
    if [ -n "$DISABLED" ] && [ -z "$RUNNING" ]; then
        exit 0
    fi
    start "$JOB"
    ;;
reload|force-reload)
    $ECHO
    $ECHO "Since the script you are attempting to invoke has been converted to an"
    $ECHO "Upstart job, you may also use the reload(8) utility, e.g. reload $JOB"
    reload "$JOB"
    ;;
*)
    $ECHO
    $ECHO "The script you are attempting to invoke has been converted to an Upstart" 1>&2
    $ECHO "job, but $COMMAND is not supported for Upstart jobs." 1>&2
    exit 1
esac
service mysql stop
stop: Unknown instance:
service mysql start
start: Job failed to start
ps -ef | grep -i mysql
root      6450  6394  0 22:05 pts/0    00:00:00 grep --color=auto -i mysql
grep -i mysql /var/log/syslog
Sep 12 20:48:07 mtcweb001 kernel: [977751.708986] type=1400 audit(1379015287.718:21): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=8707 comm="apparmor_parser"
Sep 12 20:48:07 mtcweb001 kernel: [977751.759375] init: mysql main process (8711) terminated with status 1
Sep 12 20:48:07 mtcweb001 kernel: [977751.759434] init: mysql main process ended, respawning
Sep 12 20:48:08 mtcweb001 kernel: [977752.759492] init: mysql post-start process (8712) terminated with status 1
Sep 12 20:48:08 mtcweb001 kernel: [977752.778580] type=1400 audit(1379015288.786:22): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=8735 comm="apparmor_parser"
Sep 12 20:48:08 mtcweb001 kernel: [977752.829382] init: mysql main process (8739) terminated with status 1
Sep 12 20:48:08 mtcweb001 kernel: [977752.829467] init: mysql main process ended, respawning
Sep 12 20:48:09 mtcweb001 kernel: [977753.828191] init: mysql post-start process (8740) terminated with status 1
Sep 12 20:48:09 mtcweb001 kernel: [977753.849473] type=1400 audit(1379015289.857:23): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=8763 comm="apparmor_parser"
Sep 12 20:48:09 mtcweb001 kernel: [977753.905021] init: mysql main process (8767) terminated with status 1
Sep 12 20:48:09 mtcweb001 kernel: [977753.905077] init: mysql respawning too fast, stopped
Sep 12 20:50:59 mtcweb001 kernel: [977923.035704] type=1400 audit(1379015459.024:13): apparmor="STATUS" operation="profile_load" name="/usr/sbin/mysqld" pid=953 comm="apparmor_parser"
Sep 12 20:51:00 mtcweb001 kernel: [977924.623310] type=1400 audit(1379015460.612:19): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=1026 comm="apparmor_parser"
Sep 12 20:51:05 mtcweb001 kernel: [977929.498249] init: mysql main process (1030) terminated with status 1
Sep 12 20:51:05 mtcweb001 kernel: [977929.498306] init: mysql main process ended, respawning
Sep 12 20:51:06 mtcweb001 kernel: [977930.437181] init: mysql post-start process (1032) terminated with status 1
Sep 12 20:51:06 mtcweb001 kernel: [977930.458488] type=1400 audit(1379015466.447:20): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=1242 comm="apparmor_parser"
Sep 12 20:51:06 mtcweb001 kernel: [977930.507946] init: mysql main process (1246) terminated with status 1
Sep 12 20:51:06 mtcweb001 kernel: [977930.508000] init: mysql main process ended, respawning
Sep 12 20:51:07 mtcweb001 kernel: [977931.511377] init: mysql post-start process (1247) terminated with status 1
Sep 12 20:51:07 mtcweb001 kernel: [977931.532272] type=1400 audit(1379015467.519:21): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=1287 comm="apparmor_parser"
Sep 12 20:51:07 mtcweb001 kernel: [977931.583652] init: mysql main process (1291) terminated with status 1
Sep 12 20:51:07 mtcweb001 kernel: [977931.583723] init: mysql respawning too fast, stopped
Sep 12 20:53:53 mtcweb001 kernel: [978097.688889] type=1400 audit(1379015633.652:22): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2280 comm="apparmor_parser"
Sep 12 20:53:53 mtcweb001 kernel: [978097.738975] init: mysql main process (2284) terminated with status 1
Sep 12 20:53:53 mtcweb001 kernel: [978097.739045] init: mysql main process ended, respawning
Sep 12 20:53:54 mtcweb001 kernel: [978098.748114] init: mysql post-start process (2285) terminated with status 1
Sep 12 20:53:54 mtcweb001 kernel: [978098.767137] type=1400 audit(1379015634.728:23): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2314 comm="apparmor_parser"
Sep 12 20:53:54 mtcweb001 kernel: [978098.818450] init: mysql main process (2318) terminated with status 1
Sep 12 20:53:54 mtcweb001 kernel: [978098.818507] init: mysql main process ended, respawning
Sep 12 20:53:55 mtcweb001 kernel: [978099.817846] init: mysql post-start process (2319) terminated with status 1
Sep 12 20:53:55 mtcweb001 kernel: [978099.837249] type=1400 audit(1379015635.800:24): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2342 comm="apparmor_parser"
Sep 12 20:53:55 mtcweb001 kernel: [978099.884365] init: mysql main process (2346) terminated with status 1
Sep 12 20:53:55 mtcweb001 kernel: [978099.884430] init: mysql respawning too fast, stopped
Sep 12 20:57:27 mtcweb001 kernel: [978311.560296] type=1400 audit(1379015847.499:25): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2861 comm="apparmor_parser"
Sep 12 20:57:27 mtcweb001 kernel: [978311.717120] init: mysql main process (2865) terminated with status 1
Sep 12 20:57:27 mtcweb001 kernel: [978311.717178] init: mysql main process ended, respawning
Sep 12 20:57:28 mtcweb001 kernel: [978312.612552] init: mysql post-start process (2866) terminated with status 1
Sep 12 20:57:28 mtcweb001 kernel: [978312.632940] type=1400 audit(1379015848.571:26): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2896 comm="apparmor_parser"
Sep 12 20:57:28 mtcweb001 kernel: [978312.680481] init: mysql main process (2900) terminated with status 1
Sep 12 20:57:28 mtcweb001 kernel: [978312.680589] init: mysql main process ended, respawning
Sep 12 20:57:29 mtcweb001 kernel: [978313.685133] init: mysql post-start process (2901) terminated with status 1
Sep 12 20:57:29 mtcweb001 kernel: [978313.704668] type=1400 audit(1379015849.643:27): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2930 comm="apparmor_parser"
Sep 12 20:57:29 mtcweb001 kernel: [978313.761914] init: mysql main process (2934) terminated with status 1
Sep 12 20:57:29 mtcweb001 kernel: [978313.761971] init: mysql respawning too fast, stopped
Sep 12 20:59:51 mtcweb001 kernel: [978455.908352] type=1400 audit(1379015991.823:28): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=3027 comm="apparmor_parser"
Sep 12 20:59:51 mtcweb001 kernel: [978455.958808] init: mysql main process (3031) terminated with status 1
Sep 12 20:59:51 mtcweb001 kernel: [978455.958864] init: mysql main process ended, respawning
Sep 12 20:59:52 mtcweb001 kernel: [978456.963349] init: mysql post-start process (3032) terminated with status 1
Sep 12 20:59:52 mtcweb001 kernel: [978456.982963] type=1400 audit(1379015992.899:29): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=3063 comm="apparmor_parser"
Sep 12 20:59:52 mtcweb001 kernel: [978457.033235] init: mysql main process (3067) terminated with status 1
Sep 12 20:59:52 mtcweb001 kernel: [978457.033291] init: mysql main process ended, respawning
Sep 12 20:59:53 mtcweb001 kernel: [978458.034843] init: mysql post-start process (3068) terminated with status 1
Sep 12 20:59:53 mtcweb001 kernel: [978458.054455] type=1400 audit(1379015993.971:30): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=3100 comm="apparmor_parser"
Sep 12 20:59:54 mtcweb001 kernel: [978458.105610] init: mysql main process (3104) terminated with status 1
Sep 12 20:59:54 mtcweb001 kernel: [978458.105678] init: mysql respawning too fast, stopped
root@mtcweb001:~# clear
root@mtcweb001:~# grep -i mysql /var/log/syslog
Sep 12 20:48:07 mtcweb001 kernel: [977751.708986] type=1400 audit(1379015287.718:21): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=8707 comm="apparmor_parser"
Sep 12 20:48:07 mtcweb001 kernel: [977751.759375] init: mysql main process (8711) terminated with status 1
Sep 12 20:48:07 mtcweb001 kernel: [977751.759434] init: mysql main process ended, respawning
Sep 12 20:48:08 mtcweb001 kernel: [977752.759492] init: mysql post-start process (8712) terminated with status 1
Sep 12 20:48:08 mtcweb001 kernel: [977752.778580] type=1400 audit(1379015288.786:22): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=8735 comm="apparmor_parser"
Sep 12 20:48:08 mtcweb001 kernel: [977752.829382] init: mysql main process (8739) terminated with status 1
Sep 12 20:48:08 mtcweb001 kernel: [977752.829467] init: mysql main process ended, respawning
Sep 12 20:48:09 mtcweb001 kernel: [977753.828191] init: mysql post-start process (8740) terminated with status 1
Sep 12 20:48:09 mtcweb001 kernel: [977753.849473] type=1400 audit(1379015289.857:23): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=8763 comm="apparmor_parser"
Sep 12 20:48:09 mtcweb001 kernel: [977753.905021] init: mysql main process (8767) terminated with status 1
Sep 12 20:48:09 mtcweb001 kernel: [977753.905077] init: mysql respawning too fast, stopped
Sep 12 20:50:59 mtcweb001 kernel: [977923.035704] type=1400 audit(1379015459.024:13): apparmor="STATUS" operation="profile_load" name="/usr/sbin/mysqld" pid=953 comm="apparmor_parser"
Sep 12 20:51:00 mtcweb001 kernel: [977924.623310] type=1400 audit(1379015460.612:19): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=1026 comm="apparmor_parser"
Sep 12 20:51:05 mtcweb001 kernel: [977929.498249] init: mysql main process (1030) terminated with status 1
Sep 12 20:51:05 mtcweb001 kernel: [977929.498306] init: mysql main process ended, respawning
Sep 12 20:51:06 mtcweb001 kernel: [977930.437181] init: mysql post-start process (1032) terminated with status 1
Sep 12 20:51:06 mtcweb001 kernel: [977930.458488] type=1400 audit(1379015466.447:20): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=1242 comm="apparmor_parser"
Sep 12 20:51:06 mtcweb001 kernel: [977930.507946] init: mysql main process (1246) terminated with status 1
Sep 12 20:51:06 mtcweb001 kernel: [977930.508000] init: mysql main process ended, respawning
Sep 12 20:51:07 mtcweb001 kernel: [977931.511377] init: mysql post-start process (1247) terminated with status 1
Sep 12 20:51:07 mtcweb001 kernel: [977931.532272] type=1400 audit(1379015467.519:21): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=1287 comm="apparmor_parser"
Sep 12 20:51:07 mtcweb001 kernel: [977931.583652] init: mysql main process (1291) terminated with status 1
Sep 12 20:51:07 mtcweb001 kernel: [977931.583723] init: mysql respawning too fast, stopped
Sep 12 20:53:53 mtcweb001 kernel: [978097.688889] type=1400 audit(1379015633.652:22): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2280 comm="apparmor_parser"
Sep 12 20:53:53 mtcweb001 kernel: [978097.738975] init: mysql main process (2284) terminated with status 1
Sep 12 20:53:53 mtcweb001 kernel: [978097.739045] init: mysql main process ended, respawning
Sep 12 20:53:54 mtcweb001 kernel: [978098.748114] init: mysql post-start process (2285) terminated with status 1
Sep 12 20:53:54 mtcweb001 kernel: [978098.767137] type=1400 audit(1379015634.728:23): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2314 comm="apparmor_parser"
Sep 12 20:53:54 mtcweb001 kernel: [978098.818450] init: mysql main process (2318) terminated with status 1
Sep 12 20:53:54 mtcweb001 kernel: [978098.818507] init: mysql main process ended, respawning
Sep 12 20:53:55 mtcweb001 kernel: [978099.817846] init: mysql post-start process (2319) terminated with status 1
Sep 12 20:53:55 mtcweb001 kernel: [978099.837249] type=1400 audit(1379015635.800:24): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2342 comm="apparmor_parser"
Sep 12 20:53:55 mtcweb001 kernel: [978099.884365] init: mysql main process (2346) terminated with status 1
Sep 12 20:53:55 mtcweb001 kernel: [978099.884430] init: mysql respawning too fast, stopped
Sep 12 20:57:27 mtcweb001 kernel: [978311.560296] type=1400 audit(1379015847.499:25): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2861 comm="apparmor_parser"
Sep 12 20:57:27 mtcweb001 kernel: [978311.717120] init: mysql main process (2865) terminated with status 1
Sep 12 20:57:27 mtcweb001 kernel: [978311.717178] init: mysql main process ended, respawning
Sep 12 20:57:28 mtcweb001 kernel: [978312.612552] init: mysql post-start process (2866) terminated with status 1
Sep 12 20:57:28 mtcweb001 kernel: [978312.632940] type=1400 audit(1379015848.571:26): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2896 comm="apparmor_parser"
Sep 12 20:57:28 mtcweb001 kernel: [978312.680481] init: mysql main process (2900) terminated with status 1
Sep 12 20:57:28 mtcweb001 kernel: [978312.680589] init: mysql main process ended, respawning
Sep 12 20:57:29 mtcweb001 kernel: [978313.685133] init: mysql post-start process (2901) terminated with status 1
Sep 12 20:57:29 mtcweb001 kernel: [978313.704668] type=1400 audit(1379015849.643:27): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=2930 comm="apparmor_parser"
Sep 12 20:57:29 mtcweb001 kernel: [978313.761914] init: mysql main process (2934) terminated with status 1
Sep 12 20:57:29 mtcweb001 kernel: [978313.761971] init: mysql respawning too fast, stopped
Sep 12 20:59:51 mtcweb001 kernel: [978455.908352] type=1400 audit(1379015991.823:28): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=3027 comm="apparmor_parser"
Sep 12 20:59:51 mtcweb001 kernel: [978455.958808] init: mysql main process (3031) terminated with status 1
Sep 12 20:59:51 mtcweb001 kernel: [978455.958864] init: mysql main process ended, respawning
Sep 12 20:59:52 mtcweb001 kernel: [978456.963349] init: mysql post-start process (3032) terminated with status 1
Sep 12 20:59:52 mtcweb001 kernel: [978456.982963] type=1400 audit(1379015992.899:29): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=3063 comm="apparmor_parser"
Sep 12 20:59:52 mtcweb001 kernel: [978457.033235] init: mysql main process (3067) terminated with status 1
Sep 12 20:59:52 mtcweb001 kernel: [978457.033291] init: mysql main process ended, respawning
Sep 12 20:59:53 mtcweb001 kernel: [978458.034843] init: mysql post-start process (3068) terminated with status 1
Sep 12 20:59:53 mtcweb001 kernel: [978458.054455] type=1400 audit(1379015993.971:30): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=3100 comm="apparmor_parser"
Sep 12 20:59:54 mtcweb001 kernel: [978458.105610] init: mysql main process (3104) terminated with status 1
Sep 12 20:59:54 mtcweb001 kernel: [978458.105678] init: mysql respawning too fast, stopped

This is the troubleshooting I've carried out so far and posted on Experts-Exchange!

Hope this helps!

Mon, 09/23/2013 - 08:36
Locutus

The key error seems to be this in your error.log:

/usr/sbin/mysqld: Can't create/write to file '/tmp/ibzUySNF' (Errcode: 13) 130911 22:09:16 InnoDB: Error: unable to create temporary file; errno: 13

Make sure your /tmp directory has proper permissions and, if it is on a separate partition, is mounted without the noexec option.

root@australis:~# ls -ld /tmp drwxrwxrwt 9 root root 4096 Sep 23 15:28 /tmp root@australis:~# mount [...] /dev/sdd1 on /tmp type ext4 (rw,noatime)
Thu, 09/26/2013 - 13:32
singh90

sorry for the late reply... i thought i had subscriptions set!

can you tell me what permissions need to be set? i can connect via winscp and make the amendments or ssh if required

Also, what do you mean by mounted with noexec option?

Thu, 09/26/2013 - 14:01
singh90

just re-read your post - tmp is on the same partition.

It is also permissioned 1777

Thu, 09/26/2013 - 14:19
Locutus

Okay, so directory permissions seem to be okay. Are you sure the kernel was the only thing that got upgraded when MySQL stopped working? Do you have SELinux or AppArmor installed? Those might prevent MySQL from writing to /tmp.

Thu, 09/26/2013 - 14:22
singh90

I've never installed SELinux or AppArmor - unless they come packaged with other tools?

Thu, 09/26/2013 - 15:01
Locutus

Well Ubuntu by default installs AppArmor in the 10.x and 12. server versions... So you might have it without knowing it. :)

Thu, 09/26/2013 - 17:24
singh90

just removed it :)

but more worryingly im now getting this

root@mtcweb001:~# sudo service mysql start
start: Job failed to start
root@mtcweb001:~# sudo service mysql stop
stop: Unknown instance:
Thu, 09/26/2013 - 17:53
Locutus

That's the same error you had before.

The only idea I have left right now is that you could try changing the MySQL temporary files path in my.cnf, change /tmp to e.g. /var/lib/mysql/tmp (after creating that directory with the proper owner and permissions). If MySQL has permission to write anywhere, then there.

Otherwise, did you try googling for the error messages you see? This "can't create /tmp/blabla, error 13" thing?

Thu, 09/26/2013 - 17:52
singh90

I did, i've tried everything that I've found so far :(

I read that permissions could be an issue like you suggested and even disabled logging in the cnf file to no avail.

Its looking like it might need to be a rebuild :/ just worried that I dont have a backup of the databases!

Thu, 09/26/2013 - 17:54
Locutus

I edited my previous post with the last idea I have for now.

Fri, 09/27/2013 - 04:20
singh90

Oh wow. I feel like a really really big idiot!!!!!

I've been looking at /var/tmp

I just took a look at /tmp and saw perms were set to 755 so I changed it to 1777. Now MySQL starts up!

massive facepalm