crash executing SHOW STATUS LIKE 'wsrep_%';

Run the following commands in sequence on MariaDB-Galera-server-5.5.28a

SET GLOBAL wsrep_provider="none";
SHOW STATUS LIKE 'wsrep_%';

Paused to review status and then restarted the command:

SET GLOBAL wsrep_provider="/usr/lib64/galera/libgalera_smm.so";
SHOW STATUS LIKE 'wsrep_%';

Got a crash -- see stack trace.

At the moment there was only one node in the cluster (I was trying to recover it).

130228 20:20:45 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
130228 20:20:45 [Note] WSREP: wsrep_load(): Galera 2.2(rXXXX) by Codership Oy <email address hidden> loaded succesfully.
130228 20:20:45 [Note] WSREP: Found saved state: 91e773e6-725d-11e2-0800-1d679f8c5252:-1
130228 20:20:45 [Note] WSREP: Truncating '/var/lib/mysql//galera.cache' to 134219048 bytes.
130228 20:20:45 [Note] WSREP: Passing config to GCS: base_host = 10.82.25.164; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130228 20:20:45 [Note] WSREP: Assign initial position for certification: 11379177, protocol version: -1
*** glibc detected *** /usr/sbin/mysqld: double free or corruption (out): 0x00000000019d8a60 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2abe9f0d76af]
/lib64/libc.so.6(cfree+0x4b)[0x2abe9f0db8db]
/usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM10stats_freeEP15wsrep_stats_var+0x48)[0x2abf22fb7968]
/usr/sbin/mysqld(_Z17wsrep_free_statusP3THD+0x28)[0x6993b8]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4d29)[0x595629]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x1b9)[0x596f39]
/usr/sbin/mysqld[0x59780e]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1793)[0x599d83]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x119)[0x59a309]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1f3)[0x6505a3]
/usr/sbin/mysqld(handle_one_connection+0x4c)[0x65075c]
/lib64/libpthread.so.0[0x2abe9e07183d]
/lib64/libc.so.6(clone+0x6d)[0x2abe9f13b03d]
======= Memory map: ========
00400000-00f4a000 r-xp 00000000 fd:00 603309 /usr/sbin/mysqld
01149000-01195000 rw-p 00b49000 fd:00 603309 /usr/sbin/mysqld
01195000-019de000 rw-p 01195000 00:00 0
01b94000-01c18000 rw-p 00b94000 fd:00 603309 /usr/sbin/mysqld
1e9f4000-20dc6000 rw-p 1e9f4000 00:00 0 [heap]
30a7400000-30a7540000 r-xp 00000000 fd:00 2287041 /lib64/libcrypto.so.6
30a7540000-30a7740000 ---p 00140000 fd:00 2287041 /lib64/libcrypto.so.6
30a7740000-30a7762000 rw-p 00140000 fd:00 2287041 /lib64/libcrypto.so.6
30a7762000-30a7766000 rw-p 30a7762000 00:00 0
30a7800000-30a784f000 r-xp 00000000 fd:00 2287042 /lib64/libssl.so.0.9.8x
30a784f000-30a7a4f000 ---p 0004f000 fd:00 2287042 /lib64/libssl.so.0.9.8x
30a7a4f000-30a7a55000 rw-p 0004f000 fd:00 2287042 /lib64/libssl.so.0.9.8x
3b8b000000-3b8b015000 r-xp 00000000 fd:00 1241575 /lib64/libselinux.so.1
3b8b015000-3b8b215000 ---p 00015000 fd:00 1241575 /lib64/libselinux.so.1
3b8b215000-3b8b217000 rw-p 00015000 fd:00 1241575 /lib64/libselinux.so.1
3b8b217000-3b8b218000 rw-p 3b8b217000 00:00 0
3b8b400000-3b8b43b000 r-xp 00000000 fd:00 1241573 /lib64/libsepol.so.1
3b8b43b000-3b8b63b000 ---p 0003b000 fd:00 1241573 /lib64/libsepol.so.1
3b8b63b000-3b8b63c000 rw-p 0003b000 fd:00 1241573 /lib64/libsepol.so.1
3b8b63c000-3b8b646000 rw-p 3b8b63c000 00:00 0
3b8d400000-3b8d402000 r-xp 00000000 fd:00 1241571 /lib64/libkeyutils-1.2.so
3b8d402000-3b8d601000 ---p 00002000 fd:00 1241571 /lib64/libkeyutils-1.2.so
3b8d601000-3b8d602000 rw-p 00001000 fd:00 1241571 /lib64/libkeyutils-1.2.so
3b8dc00000-3b8dc24000 r-xp 00000000 fd:00 605969 /usr/lib64/libk5crypto.so.3.1
3b8dc24000-3b8de23000 ---p 00024000 fd:00 605969 /usr/lib64/libk5crypto.so.3.1
3b8de23000-3b8de25000 rw-p 00023000 fd:00 605969 /usr/lib64/libk5crypto.so.3.1
3b8e000000-3b8e02c000 r-xp 00000000 fd:00 605971 /usr/lib64/libgssapi_krb5.so.2.2
3b8e02c000-3b8e22c000 ---p 0002c000 fd:00 605971 /usr/lib64/libgssapi_krb5.so.2.2
3b8e22c000-3b8e22e000 rw-p 0002c000 fd:00 605971 /usr/lib64/libgssapi_krb5.so.2.2
3b8e400000-3b8e408000 r-xp 00000000 fd:00 605968 /usr/lib64/libkrb5support.so.0.1
3b8e408000-3b8e607000 ---p 00008000 fd:00 605968 /usr/lib64/libkrb5support.so.0.1
3b8e607000-3b8e608000 rw-p 00007000 fd:00 605968 /usr/lib64/libkrb5support.so.0.1
3b8ec00000-3b8ec91000 r-xp 00000000 fd:00 605970 /usr/lib64/libkrb5.so.3.3
3b8ec91000-3b8ee91000 ---p 00091000 fd:00 605970 /usr/lib64/libkrb5.so.3.3
3b8ee91000-3b8ee95000 rw-p 00091000 fd:00 605970 /usr/lib64/libkrb5.so.3.3
2abe9de4d000-2abe9de69000 r-xp 00000000 fd:00 1895686 /lib64/ld-2.5.so
2abe9de69000-2abe9de6b000 rw-p 2abe9de69000 00:00 0
2abe9e069000-2abe9e06a000 r--p 0001c000 fd:00 1895686 /lib64/ld-2.5.so
2abe9e06a000-2abe9e06b000 rw-p 0001d000 fd:00 1895686 /lib64/ld-2.5.so
2abe9e06b000-2abe9e081000 r-xp 00000000 fd:00 1241569 /lib64/libpthread-2.5.so
2abe9e081000-2abe9e281000 ---p 00016000 fd:00 1241569 /lib64/libpthread-2.5.so
2abe9e281000-2abe9e282000 r--p 00016000 fd:00 1241569 /lib64/libpthread-2.5.so
2abe9e282000-2abe9e283000 rw-p 00017000 fd:00 1241569 /lib64/libpthread-2.5.so
2abe9e283000-2abe9e287000 rw-p 2abe9e283000 00:00 0
2abe9e287000-2abe9e29b000 r-xp 00000000 fd:00 1241584 /lib64/libz.so.1.2.3
2abe9e29b000-2abe9e49a000 ---p 00014000 fd:00 1241584 /lib64/libz.so.1.2.3
2abe9e49a000-2abe9e49b000 rw-p 00013000 fd:00 1241584 /lib64/libz.so.1.2.3
2abe9e49b000-2abe9e4a2000 r-xp 00000000 fd:00 1241872 /lib64/librt-2.5.so
2abe9e4a2000-2abe9e6a2000 ---p 00007000 fd:00 1241872 /lib64/librt-2.5.so
2abe9e6a2000-2abe9e6a3000 r--p 00007000 fd:00 1241872 /lib64/librt-2.5.so
2abe9e6a3000-2abe9e6a4000 rw-p 00008000 fd:00 1241872 /lib64/librt-2.5.so
2abe9e6a4000-2abe9e6a5000 rw-p 2abe9e6a4000 00:00 0
2abe9e6a5000-2abe9e6ae000 r-xp 00000000 fd:00 1241549 /lib64/libcrypt-2.5.so
2abe9e6ae000-2abe9e8ad000 ---p 00009000 fd:00 1241549 /lib64/libcrypt-2.5.so
2abe9e8ad000-2abe9e8ae000 r--p 00008000 fd:00 1241549 /lib64/libcrypt-2.5.so
2abe9e8ae000-2abe9e8af000 rw-p 00009000 fd:00 1241549 /lib64/libcrypt-2.5.so
2abe9e8af000-2abe9e8dd000 rw-p 2abe9e8af000 00:00 0
2abe9e8dd000-2abe9e8df000 r-xp 00000000 fd:00 1241864 /lib64/libdl-2.5.so
2abe9e8df000-2abe9eadf000 ---p 00002000 fd:00 1241864 /lib64/libdl-2.5.so
2abe9eadf000-2abe9eae0000 r--p 00002000 fd:00 1241864 /lib64/libdl-2.5.so
2abe9eae0000-2abe9eae1000 rw-p 00003000 fd:00 1241864 /lib64/libdl-2.5.so
2abe9eae1000-2abe9eae2000 rw-p 2abe9eae1000 00:00 0
2abe9eae2000-2abe9ebc8000 r-xp 00000000 fd:00 595078 /usr/lib64/libstdc++.so.6.0.8
2abe9ebc8000-2abe9edc7000 ---p 000e6000 fd:00 595078 /usr/lib64/libstdc++.so.6.0.8
2abe9edc7000-2abe9edcd000 r--p 000e5000 fd:00 595078 /usr/lib64/libstdc++.so.6.0.8
2abe9edcd000-2abe9edd0000 rw-p 000eb000 fd:00 595078 /usr/lib64/libstdc++.so.6.0.8
2abe9edd0000-2abe9ede2000 rw-p 2abe9edd0000 00:00 0
2abe9ede2000-2abe9ee64000 r-xp 00000000 fd:00 1241866 /lib64/libm-2.5.so
2abe9ee64000-2abe9f063000 ---p 00082000 fd:00 1241866 /lib64/libm-2.5.so
2abe9f063000-2abe9f064000 r--p 00081000 fd:00 1241866 /lib64/libm-2.5.so
2abe9f064000-2abe9f065000 rw-p 00082000 fd:00 1241866 /lib64/libm-2.5.so
2abe9f065000-2abe9f066000 rw-p 2abe9f065000 00:00 0
2abe9f066000-2abe9f1b5000 r-xp 00000000 fd:00 1241545 /lib64/libc-2.5.so
2abe9f1b5000-2abe9f3b5000 ---p 0014f000 fd:00 1241545 /lib64/libc-2.5.so
2abe9f3b5000-2abe9f3b9000 r--p 0014f000 fd:00 1241545 /lib64/libc-2.5.so
2abe9f3b9000-2abe9f3ba000 rw-p 00153000 fd:00 1241545 /lib64/libc-2.5.so
2abe9f3ba000-2abe9f3c0000 rw-p 2abe9f3ba000 00:00 0
2abe9f3c0000-2abe9f3c2000 r-xp 00000000 fd:00 1241722 /lib64/libcom_err.so.2.1
2abe9f3c2000-2abe9f5c1000 ---p 00002000 fd:00 1241722 /lib64/libcom_err.so.2.1
2abe9f5c1000-2abe9f5c2000 rw-p 00001000 fd:00 1241722 /lib64/libcom_err.so.2.1
2abe9f5c2000-2abe9f5cf000 r-xp 00000000 fd:00 1895684 /lib64/libgcc_s-4.1.2-20080825.so.1
2abe9f5cf000-2abe9f7cf000 ---p 0000d000 fd:00 1895684 /lib64/libgcc_s-4.1.2-20080825.so.1
2abe9f7cf000-2abe9f7d0000 rw-p 0000d000 fd:00 1895684 /lib64/libgcc_s-4.1.2-20080825.so.1
2abe9f7d0000-2abe9f7d1000 rw-p 2abe9f7d0000 00:00 0
2abe9f7d1000-2abe9f7e2000 r-xp 00000000 fd:00 1241871 /lib64/libresolv-2.5.so
2abe9f7e2000-2abe9f9e2000 ---p 00011000 fd:00 1241871 /lib64/libresolv-2.5.so
2abe9f9e2000-2abe9f9e3000 r--p 00011000 fd:00 1241871 /lib64/libresolv-2.5.so
2abe9f9e3000-2abe9f9e4000 rw-p 00012000 fd:00 1241871 /lib64/libresolv-2.5.so
2abe9f9e4000-2abe9f9ea000 rw-p 2abe9f9e4000 00:00 0
2abe9f9ea000-2abe9f9eb000 ---p 2abe9f9ea000 00:00 0
2abe9f9eb000-2abea03eb000 rw-p 2abe9f9eb000 00:00 0
2abea03eb000-2abea0420000 r--s 00000000 fd:00 784372 /var/db/nscd/passwd
2abea0420000-2abea0455000 r--s 00000000 fd:00 784374 /var/db/nscd/group
2abea0455000-2abec95ae000 rw-p 2abea0455000 00:00 0
2abecc000000-2abecffbe000 rw-p 2abecc000000 00:00 0
2abecffbe000-2abed0000000 ---p 2abecffbe000 00:00 0
2abed0000000-2abf1376d000 rw-p 2abed0000000 00:00 0
2abf14000000-2abf153ff000 rw-p 2abf14000000 00:00 0
2abf153ff000-2abf18000000 ---p 2abf153ff000 00:00 0
2abf18000000-2abf18001000 ---p 2abf18000000 00:00 0
2abf18001000-2abf18a01000 rw-p 2abf18001000 00:00 0
2abf18a01000-2abf18a02000 ---p 2abf18a01000 00:00 0
2abf18a02000-2abf19402000 rw-p 2abf18a02000 00:00 0
2abf19402000-2abf19403000 ---p 2abf19402000 00:00 0
2abf19403000-2abf19e03000 rw-p 2abf19403000 00:00 0
2abf19e03000-2abf19e04000 ---p 2abf19e03000 00:00 0
2abf19e04000-2abf1a804000 rw-p 2abf19e04000 00:00 0
2abf1a804000-2abf1a805000 ---p 2abf1a804000 00:00 0
2abf1a805000-2abf1b205000 rw-p 2abf1a805000 00:00 0
2abf1b205000-2abf1b206000 ---p 2abf1b205000 00:00 0
2abf1b206000-2abf1bc06000 rw-p 2abf1b206000 00:00 0
2abf1bc06000-2abf1bc07000 ---p 2abf1bc06000 00:00 0
2abf1bc07000-2abf1c607000 rw-p 2abf1bc07000 00:00 0
2abf1c607000-2abf1c608000 ---p 2abf1c607000 00:00 0
2abf1c608000-2abf1d008000 rw-p 2abf1c608000 00:00 0
2abf1d008000-2abf1d009000 ---p 2abf1d008000 00:00 0
2abf1d009000-2abf1da09000 rw-p 2abf1d009000 00:00 0
2abf1da09000-2abf1da0a000 ---p 2abf1da09000 00:00 0
2abf1da0a000-2abf1e40a000 rw-p 2abf1da0a000 00:00 0
2abf1e40a000-2abf1e40b000 ---p 2abf1e40a000 00:00 0
2abf1e40b000-2abf1ee0b000 rw-p 2abf1e40b000 00:00 0
2abf1f4ec000-2abf1fb2c000 rw-p 2abf1f4ec000 00:00 0
2abf1fb2c000-2abf1fb2d000 ---p 2abf1fb2c000 00:00 0
2abf1fb2d000-2abf2052d000 rw-p 2abf1fb2d000 00:00 0
2abf2052d000-2abf2052e000 ---p 2abf2052d000 00:00 0
2abf2052e000-2abf20f2e000 rw-p 2abf2052e000 00:00 0
2abf20f2e000-2abf20f2f000 ---p 2abf20f2e000 00:00 0
2abf20f2f000-2abf2192f000 rw-p 2abf20f2f000 00:00 0
2abf2192f000-2abf21930000 ---p 2abf2192f000 00:00 0
2abf21930000-2abf22330000 rw-p 2abf21930000 00:00 0
2abf22330000-2abf22331000 ---p 2abf22330000 00:00 0
2abf22331000-2abf22d31000 rw-p 2abf22331000 00:00 0
2abf22d31000-2abf22d32000 ---p 2abf22d31000 00:00 0
2abf22d32000-2abf22d7a000 rw-p 2abf22d32000 00:00 0
2abf22d7a000-2abf22d7b000 ---p 2abf22d7a000 00:00 0
2abf22d7b000-2abf22dc3000 rw-p 2abf22d7b000 00:00 0
2abf22dc3000-2abf22dc4000 ---p 2abf22dc3000 00:00 0
2abf22dc4000-2abf22e0c000 rw-p 2abf22dc4000 00:00 0
2abf22e0c000-2abf23025000 r-xp 00000000 fd:00 882968 /usr/lib64/galera/libgalera_smm.so
2abf23025000-2abf23224000 ---p 00219000 fd:00 882968 /usr/lib64/galera/libgalera_smm.so
2abf23224000-2abf2322f000 rw-p 00218000 fd:00 882968 /usr/lib64/galera/libgalera_smm.so
2abf2322f000-2abf23230000 rw-p 2abf2322f000 00:00 0
2abf23230000-2abf2b231000 rw-s 00000000 fd:00 1013094 /var/lib/mysql/galera.cache
2abf2b231000-2abf2b232000 rw-p 2abf2b231000 00:00 0
2abf63231000-2abf63232000 ---p 2abf63231000 00:00 0
2abf63232000-2abf63c32000 rw-p 2abf63232000 00:00 0
2abf63c33000-2abf63c34000 ---p 2abf63c33000 00:00 0
2abf63c34000-2abf63c7c000 rw-p 2abf63c34000 00:00 0
2abf63c7c000-2abf63c7d000 ---p 2abf63c7c000 00:00 0
2abf63c7d000-2abf63cc5000 rw-p 2abf63c7d000 00:00 0
7fff78073000-7fff78088000 rw-p 7ffffffe9000 00:00 0 [stack]
7fff78197000-7fff7819a000 r-xp 7fff78197000 00:00 0 [vdso]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vsyscall]
130228 20:20:46 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 5.5.28a-MariaDB-log
key_buffer_size=536870912
read_buffer_size=131072
max_used_connections=4
max_threads=502
thread_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 724320 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x1eb99290
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x2abf22dc20a8 thread_stack 0x48000
??:0(my_print_stacktrace)[0xa9a3ae]
??:0(handle_fatal_signal)[0x6e383b]
??:0(??)[0x2abe9e079ca0]
??:0(??)[0x2abe9f0962c5]
??:0(??)[0x2abe9f097d70]
??:0(??)[0x2abe9f0cfb4b]
??:0(??)[0x2abe9f0d76af]
??:0(??)[0x2abe9f0db8db]
??:0(??)[0x2abf22fb7968]
??:0(wsrep_free_status(THD*))[0x6993b8]
??:0(mysql_execute_command(THD*))[0x595629]
??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x596f39]
??:0(_ZL17wsrep_mysql_parseP3THDPcjP12Parser_state)[0x59780e]
??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x599d83]
??:0(do_command(THD*))[0x59a309]
??:0(do_handle_one_connection(THD*))[0x6505a3]
??:0(handle_one_connection)[0x65075c]
??:0(??)[0x2abe9e07183d]
??:0(??)[0x2abe9f13b03d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x1ebe3738): SHOW STATUS LIKE 'wsrep_%'

Connection ID (thread ID): 1
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=on,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
130228 20:20:46 mysqld_safe Number of processes running now: 0
130228 20:20:46 mysqld_safe mysqld restarted
130228 20:20:46 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.XStWgB5118
130228 20:20:49 mysqld_safe WSREP: Recovered position 91e773e6-725d-11e2-0800-1d679f8c5252:11379177
130228 20:20:49 [Note] WSREP: wsrep_start_position var submitted: '91e773e6-725d-11e2-0800-1d679f8c5252:11379177'