惊心动魄-生产环境Mysql启动失败
一、问题描述
- 1、客户数据库从服务器,套接字丢失/tmp/mysql.sock,导致mysql 直连本地提示访问拒绝
- 2、正常解决方法:需要人为重启动数据库。
- 3、重启动过程出现服务器不能启动(惊悚……)直接提示bug不能启动。
#二、处理流程
- 1、检查mysql 日志(日志如下) 按照错误提示修改key_buffer_size这几个参数,启动依旧失败。
021-06-05 14:27:07 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-06-05 14:27:07 25403 [Note] Plugin 'FEDERATED' is disabled.
06:27:07 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=4194304
read_buffer_size=4194304
max_used_connections=0
max_threads=3000
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 24621158 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7ffff99f8040
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ffff99fb700 thread_stack 0x80000
/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x35)[0x9024f5]
/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x3e8)[0x66f518]
/lib64/libpthread.so.0[0x3db500f7e0]
/usr/local/mysql/bin/mysqld(_Z18ha_resolve_by_nameP3THDPK19st_mysql_lex_stringb+0x70)[0x58db80]
/usr/local/mysql/bin/mysqld(_Z14open_table_defP3THDP11TABLE_SHAREj+0x2dcd)[0x787b3d]
/usr/local/mysql/bin/mysqld(_Z15get_table_shareP3THDP10TABLE_LISTPKcjjPij+0x189)[0x69dec9]
/usr/local/mysql/bin/mysqld[0x6a049c]
/usr/local/mysql/bin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP18Open_table_context+0x846)[0x6a6966]
/usr/local/mysql/bin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0xbdc)[0x6a9ebc]
/usr/local/mysql/bin/mysqld(_Z20open_and_lock_tablesP3THDP10TABLE_LISTbjP19Prelocking_strategy+0x4e)[0x6aa10e]
/usr/local/mysql/bin/mysqld(_Z11plugin_initPiPPci+0xf32)[0x6fef52]
/usr/local/mysql/bin/mysqld[0x582fa0]
/usr/local/mysql/bin/mysqld(_Z11mysqld_mainiPPc+0x375)[0x583e55]
/lib64/libc.so.6(__libc_start_main+0x100)[0x3db4c1ed20]
/usr/local/mysql/bin/mysqld(__gxx_personality_v0+0x2d9)[0x578469]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 0
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2021-06-05T06:27:07.478146Z mysqld_safe mysqld from pid file /data/mysql/mysqld.pid ended
- 2、咨询客户发现,这台机器原先是mysql5.6 ,后来升级至5.7(采用的是二进制安装)
mysql5.6安装路径:/usr/local/mysql
mysql5.7安装路径:/usr/local/mysql-5.7.16-linux-glibc2.5-x86_64
根据上面异常信息,错误信息依旧是连接/usr/local/mysql。所以我修改/etc/profile的环境变量
export PATH=.$PATH:/usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/bin
启动数据库错误问题依旧。。。。。。
/usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqld_safe --defaults-file=/etc/my.cnf --user=mysql &
三、解决方法
根据提示:mysql5.7的mysqld_safe启动时,引用的mysql5.6动态库,是不是因为环境变量之类有影响。直接到/usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/ 路径下启动试试,启动成功^-^。
cd /usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/
/bin/sh ./bin/mysqld_safe --defaults-file=/etc/my.cnf --user=mysql &
花絮:如果启动不起来,我得重新作主从,估计要花费7-8小时。(影响客户使用不了从库)。幸亏顺利启动了~~。
[root@prod-slave mysql]# ps aux |grep mysql
root 6465 0.0 0.0 103328 852 pts/8 S+ 15:15 0:00 grep mysql
root 30126 0.0 0.0 106252 1424 pts/11 S+ 15:12 0:00 /bin/sh ./bin/mysqld_safe --defaults-file=/etc/my.cnf --user=mysql
mysql 30848 20.9 9.6 21781692 3160368 pts/11 Sl+ 15:12 0:38 /usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqld --defaults-file=/etc/my.cnf --basedir=/usr/local/mysql-5.7.16-linux-glibc2.5-x86_64 --datadir=/data/mysql/data/ --plugin-dir=/usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/lib/plugin --user=mysql --log-error=/data/logs/mysql/error.log --open-files-limit=10240 --pid-file=/data/mysql/mysqld.pid --socket=/data/mysql/mysql.sock --port=3306
[root@prod-slave mysql]# lsof -p 30126
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sh 30126 root cwd DIR 252,1 4096 1310755 /usr/local/mysql-5.7.16-linux-glibc2.5-x86_64
sh 30126 root rtd DIR 252,1 4096 2 /
sh 30126 root txt REG 252,1 942200 1180875 /bin/bash
sh 30126 root mem REG 252,1 161776 262169 /lib64/ld-2.12.so
sh 30126 root mem REG 252,1 1930416 267634 /lib64/libc-2.12.so
sh 30126 root mem REG 252,1 23088 267913 /lib64/libdl-2.12.so
sh 30126 root mem REG 252,1 134792 267872 /lib64/libtinfo.so.5.7
sh 30126 root mem REG 252,1 99174448 1071533 /usr/lib/locale/locale-archive
sh 30126 root mem REG 252,1 26060 1053259 /usr/lib64/gconv/gconv-modules.cache
sh 30126 root 0u CHR 136,11 0t0 14 /dev/pts/11
sh 30126 root 1u CHR 136,11 0t0 14 /dev/pts/11
sh 30126 root 2u CHR 136,11 0t0 14 /dev/pts/11
sh 30126 root 255r REG 252,1 27186 1324136 /usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqld_safe
[root@prod-slave mysql]# lsof -p 30848
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mysqld 30848 mysql cwd DIR 252,17 4096 3932162 /data/mysql/data
mysqld 30848 mysql rtd DIR 252,1 4096 2 /
mysqld 30848 mysql txt REG 252,1 253303409 1324134 /usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqld
mysqld 30848 mysql mem REG 252,1 5624 270897 /lib64/libaio.so.1.0.1
mysqld 30848 mysql mem REG 252,1 161776 262169 /lib64/ld-2.12.so
mysqld 30848 mysql mem REG 252,1 1930416 267634 /lib64/libc-2.12.so
mysqld 30848 mysql mem REG 252,1 146592 267906 /lib64/libpthread-2.12.so
mysqld 30848 mysql mem REG 252,1 23088 267913 /lib64/libdl-2.12.so
mysqld 30848 mysql mem REG 252,1 47760 267909 /lib64/librt-2.12.so
mysqld 30848 mysql mem REG 252,1 600048 262160 /lib64/libm-2.12.so
mysqld 30848 mysql mem REG 252,1 93320 267908 /lib64/libgcc_s-4.4.7-20120601.so.1
mysqld 30848 mysql mem REG 252,1 43928 267920 /lib64/libcrypt-2.12.so
mysqld 30848 mysql mem REG 252,1 12776 267919 /lib64/libfreebl3.so
mysqld 30848 mysql mem REG 252,1 217016 786460 /var/db/nscd/hosts
mysqld 30848 mysql mem REG 252,1 66432 267620 /lib64/libnss_files-2.12.so
mysqld 30848 mysql mem REG 252,1 989840 1071530 /usr/lib64/libstdc++.so.6.0.13
mysqld 30848 mysql 0r CHR 1,3 0t0 3824 /dev/null
mysqld 30848 mysql 1w REG 252,17 12824 262147 /data/logs/mysql/error.log
mysqld 30848 mysql 2w REG 252,17 12824 262147 /data/logs/mysql/error.log
mysqld 30848 mysql 3u REG 252,17 62 262163 /data/logs/mysql/binlog.index
mysqld 30848 mysql 4uW REG 252,17 6777995264 3932167 /data/mysql/data/ibdata1
mysqld 30848 mysql 5u REG 252,1 0 396354 /tmp/ibk9eQtl (deleted)
mysqld 30848 mysql 6u REG