nmap causes MySQL crash

Bug #1184034 reported by Kate Cronin
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Galera
Fix Released
High
Teemu Ollakka
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Fix Released
Undecided
Unassigned

Bug Description

Basic info:

# uname -a
Linux 2.6.32-235.el6.x86_64 #1 SMP Fri Feb 17 11:48:53 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa | grep -i percona
percona-xtrabackup-2.0.7-552.rhel6.x86_64
Percona-XtraDB-Cluster-shared-5.5.30-23.7.4.406.rhel6.x86_64
percona-toolkit-2.1.4-1.noarch
Percona-XtraDB-Cluster-client-5.5.30-23.7.4.406.rhel6.x86_64
Percona-Server-shared-compat-5.5.30-rel30.2.508.rhel6.x86_64
Percona-XtraDB-Cluster-server-5.5.30-23.7.4.406.rhel6.x86_64
Percona-XtraDB-Cluster-galera-2.5-1.150.rhel6.x86_64

I can quite reliably crash MySQL by running 'nmap' against the server running it, but only as a non-root user. If I run nmap as root, all's well. I think the difference lies in how nmap scans ports when run as root vs. non-root. Some strace output:

as root, it uses "sendto()":
sendto(3, "E\0\0,\260x\0\0(\6\211#\212H\204\320\212H\277\317\364v\21\327\261\"?\313\0\0\0\0"..., 44, 0, {sa_family=AF_INET, sin_port=htons(4567), sin_addr=inet_addr("xxx.xxx.xxx.xxx")}, 16) = 44

as non-root, it uses "connect()":
connect(3, {sa_family=AF_INET, sin_port=htons(4567), sin_addr=inet_addr("xxx.xxx.xxx.xxx")}, 16) = -1 EINPROGRESS (Operation now in progress)

(I don't know that it's the connection to port 4567 that is the culprit; just used those lines from strace output as examples).

MySQL dies like so:

terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<asio::system_error> >'
  what(): Transport endpoint is not connected
22:47:10 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=11
max_threads=2002
thread_count=9
connection_count=9
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4389869 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7d0765]
/usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x6a9c64]
/lib64/libpthread.so.0(+0xf520)[0x7f0ddfe05520]
/lib64/libc.so.6(gsignal+0x35)[0x7f0dde9dba45]
/lib64/libc.so.6(abort+0x175)[0x7f0dde9dd225]
/usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x12d)[0x7f0ddc13ca7d]
/usr/lib64/libstdc++.so.6(+0xbcc06)[0x7f0ddc13ac06]
/usr/lib64/libstdc++.so.6(+0xbcc33)[0x7f0ddc13ac33]
/usr/lib64/libstdc++.so.6(+0xbcd2e)[0x7f0ddc13ad2e]
/usr/lib64/libgalera_smm.so(_ZN5boost15throw_exceptionIN4asio12system_errorEEEvRKT_+0x18a)[0x7f0ddc4a93ca]
/usr/lib64/libgalera_smm.so(_ZN4asio6detail11throw_errorERKNS_10error_codeE+0x5b)[0x7f0ddc4a94eb]
/usr/lib64/libgalera_smm.so(_ZNK4asio12basic_socketINS_2ip3tcpENS_21stream_socket_serviceIS2_EEE15remote_endpointEv+0xc1)[0x7f0ddc4a9671]
/usr/lib64/libgalera_smm.so(_ZN5gcomm13AsioTcpSocket18assign_remote_addrEv+0x5e7)[0x7f0ddc49a657]
/usr/lib64/libgalera_smm.so(_ZN5gcomm15AsioTcpAcceptor14accept_handlerEN5boost10shared_ptrINS_6SocketEEERKN4asio10error_codeE+0xc4)[0x7f0ddc49b0e4]
/usr/lib64/libgalera_smm.so(_ZN4asio6detail25reactive_socket_accept_opINS_12basic_socketINS_2ip3tcpENS_21stream_socket_serviceIS4_EEEES4_N5boost3_bi6bind_tIvNS8_4_mfi3mf2IvN5gcomm15AsioTcpAcceptorENS8_10shared_ptrINSD_6SocketEEERKNS_10error_codeEEENS9_5list3INS9_5valueIPSE_EENSN_ISH_EEPFNS8_3argILi1EEEvEEEEEE11do_completeEPNS0_15task_io_serviceEPNS0_25task_io_service_operationESI_m+0x180)[0x7f0ddc4a4eb0]
/usr/lib64/libgalera_smm.so(_ZN4asio6detail15task_io_service3runERNS_10error_codeE+0x459)[0x7f0ddc4c68c9]
/usr/lib64/libgalera_smm.so(_ZN5gcomm12AsioProtonet10event_loopERKN2gu8datetime6PeriodE+0x1d6)[0x7f0ddc4be816]
/usr/lib64/libgalera_smm.so(_ZN9GCommConn3runEv+0x57)[0x7f0ddc4d8777]
/usr/lib64/libgalera_smm.so(_ZN9GCommConn6run_fnEPv+0x9)[0x7f0ddc4dd469]
/lib64/libpthread.so.0(+0x77e1)[0x7f0ddfdfd7e1]
/lib64/libc.so.6(clone+0x6d)[0x7f0ddea8f8ed]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
130524 15:47:10 mysqld_safe Number of processes running now: 0
130524 15:47:10 mysqld_safe WSREP: not restarting wsrep node automatically
130524 15:47:10 mysqld_safe mysqld from pid file /mnt/ssd/mysql/xxx.pid ended

I saw this discussion of a very similar problem on the "Percona discussions" google group, but I'm not sure an actual bug was ever submitted - a search on this site didn't find anything, but I apologize if this ends up being a duplicate.

https://groups.google.com/forum/?fromgroups#!topic/percona-discussion/gUWvXOKR_88

Thanks!

description: updated
affects: percona-server → percona-xtradb-cluster
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Seems to be related to https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1153727 .. you can follow updates in both places.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

I can reproduce this.

However, we strongly recommend that cluster ports be secured externally as well with iptables/pf since otherwise other ports of cluster are vulnerable as well - like anyone will be able to SST from a donor since no checks are performed there against any whitelist/blacklist.

But, again, this shouldn't crash the server. It is being investigated in lp:1153727.

Changed in percona-xtradb-cluster:
status: New → Confirmed
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

I can also reproduce that with/without root the behaviour is different wrt. crash.

Revision history for this message
Laurent Minost (lolomin) wrote :

Hi Raghavendra,

Thanks for these informations !
I've faced the same crash on one of my development PXDBC node and the interrogation is : what can be done exactly to avoid this bug ? Firewalling of ports 4444 and 4567 from IP others than the one used by the cluster nodes should be enough ?
To better understand the problem what will be done on your side to fix this, what is the root cause of this sudden crash ?

Regards,

Laurent

Changed in galera:
assignee: nobody → Teemu Ollakka (teemu-ollakka)
importance: Undecided → High
milestone: none → 23.2.6
status: New → Confirmed
Changed in percona-xtradb-cluster:
milestone: none → 5.5.31-24.8
Changed in galera:
status: Confirmed → In Progress
Revision history for this message
Laurent Minost (lolomin) wrote :

Hi,

Raghavendra D Prabhu (raghavendra-prabhu) wrote on 2013-05-25: #2
"I can reproduce this.

However, we strongly recommend that cluster ports be secured externally as well with iptables/pf since otherwise other ports of cluster are vulnerable as well - like anyone will be able to SST from a donor since no checks are performed there against any whitelist/blacklist."

Could you please precise this behavior ? Is it really possible presently to get a complete SST from an outside attacker which gained access to the LAN on which a cluster is located when this cluster uses xtrabackup as SST method ? I mean : there is no authentication (like wsrep_sst_auth) used between joiner and donor ?

Thanks !

Regards,

Laurent

Revision history for this message
Alex Yurchenko (ayurchen) wrote :

Laurent,

1) It depends on your SST script. E.g. mysqldump uses authentication, rsync does not. So of course if rsync daemon is running on your node, and it is accessible from the insecure network, then indeed, an attacker can get a snapshot of your data directory.

2) For authentication to really be of any protection, it (and data transfer) must be performed over a well-encrypted channel. So speaking about authentication without encryption is moot. And so far in the encryption department people trust VPN or SSH/SSL tunnels. So you know what to do - if you are in insecure network.

Regards,
Alex

Revision history for this message
Alex Yurchenko (ayurchen) wrote :

fix committed in r152

Changed in galera:
status: In Progress → Fix Committed
Changed in percona-xtradb-cluster:
status: Confirmed → Fix Released
Changed in galera:
status: Fix Committed → Fix Released
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1360

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.