MySQL patches by Codership

Slave thread crash during FK checking

Bug #1204361 reported by Seppo Jaakola
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
Fix Released
Medium
Seppo Jaakola
MySQL patches by Codership 5.5.33-23.7.6

Bug Description

PXC 5.5.30 node crashed with following trace:

Thread pointer: 0x7fc28c000990
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fc2fbb26db8 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7d0765]
/usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x6a9c64]
/lib64/libpthread.so.0[0x31b580f500]
/usr/sbin/mysqld[0x8df0bc]
/usr/sbin/mysqld[0x84a967]
/usr/sbin/mysqld[0x91b61b]
/usr/sbin/mysqld[0x91c28f]
/usr/sbin/mysqld[0x824863]
/usr/sbin/mysqld[0x825665]
/usr/sbin/mysqld[0x8268fa]
/usr/sbin/mysqld[0x80e816]
/usr/sbin/mysqld[0x91ab2c]
/usr/sbin/mysqld[0x91bf9b]
/usr/sbin/mysqld[0x824dc3]
/usr/sbin/mysqld[0x8260fc]
/usr/sbin/mysqld[0x826741]
/usr/sbin/mysqld[0x811611]
/usr/sbin/mysqld[0x7ed68f]
/usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x5e)[0x6ae86e]
/usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEPK14Relay_log_info+0x148)[0x74d6c8]
/usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0x22f)[0x75285f]
/usr/sbin/mysqld[0x59633e]
/usr/sbin/mysqld(_Z14wsrep_apply_cbPvPKvml+0x8c)[0x59699c]
/usr/lib64/libgalera_smm.so(+0x194c2d)[0x7fc8551f4c2d]
/usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM9apply_trxEPvPNS_9TrxHandleE+0x223)[0x7fc8551f5b83]
/usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM11process_trxEPvPNS_9TrxHandleE+0x45)[0x7fc8551f64d5]
/usr/lib64/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_action+0x2bc)[0x7fc8551d1e0c]
/usr/lib64/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPv+0x58)[0x7fc8551d22c8]
/usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0x7d)[0x7fc8551ede0d]
/usr/lib64/libgalera_smm.so(galera_recv+0x23)[0x7fc855206c13]
/usr/sbin/mysqld(_Z25wsrep_replication_processP3THD+0x4b)[0x595edb]
/usr/sbin/mysqld(start_wsrep_THD+0x3ad)[0x51e34d]
/lib64/libpthread.so.0[0x31b5807851]

The upper innodb part was resolved as:

0x8df0bc lock_number_of_rows_locked + 76
0x84a967 trx_print + 327
0x91b61b row_ins_foreign_report_add_err + 251
0x91c28f row_ins_check_foreign_constraint + 2127
0x824863 wsrep_row_upd_check_foreign_constraints + 675
0x825665 row_upd_sec_index_entry + 1509
0x8268fa row_upd_step + 906
0x80e816 row_update_cascade_for_mysql + 86
0x91ab2c row_ins_foreign_check_on_constraint + 2492
0x91bf9b row_ins_check_foreign_constraint + 1371
0x824dc3 row_upd_check_references_constraints + 675
0x8260fc row_upd_clust_step + 2476
0x826741 row_upd_step + 465
0x811611 row_update_for_mysql + 289
0x7ed68f _ZN11ha_innobase10delete_rowEPKh + 159

Probable cause for this crash is inconsistency that was introduced with earlier DDL operations. However, this stack trace shows that slave applier does excessive foreign key checks, which should be optimized out.

Seppo Jaakola (seppo-jaakola)
Changed in codership-mysql:
importance: Undecided → Medium
assignee: nobody → Seppo Jaakola (seppo-jaakola)
milestone: none → 5.5.32-23.7.6
Revision history for this message
Seppo Jaakola (seppo-jaakola) wrote :

Fix for skipping FK parent row tracking in replication slave processing was pushed in as part of 5.5.32 merge: http://bazaar.launchpad.net/~codership/codership-mysql/5.5-23/revision/3894

The actual patch is in storage/innobase/row0upd.c

But, this patch needs some fixing to be buildable, here is the last commit for this issue: http://bazaar.launchpad.net/~codership/codership-mysql/5.5-23/revision/3896

Changed in codership-mysql:
status: New → In Progress
status: In Progress → Fix Committed
Alex Yurchenko (ayurchen)
Changed in codership-mysql:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.