MySQL--MHA与GTID

View Code

通过mysqldump进行备份

通过mysqldump做一个全量备份

[[email protected] ~]# mysqldump --all-databases --single-transaction --routines --events --host=127.0.0.1 --port=3306 --user=root > dump.sql 

生成的dump.sql文件里包含了设置gtid_purged的语句

dump.sql:

SET @MYSQLDUMP_TEMP_LOG_BIN = @@SESSION.SQL_LOG_BIN;SET @@SESSION.SQL_LOG_BIN= 0;...SET @@GLOBAL.GTID_PURGED='e10c75be-5c1b-11e6-ab7c-000c296078ae:1-10';...SET @@SESSION.SQL_LOG_BIN = @MYSQLDUMP_TEMP_LOG_BIN; 

恢复数据前需要先通过reset master清空gtid_executed变量

[[email protected] ~]# mysql -h127.1 -e 'reset master'[[email protected] ~]# mysql -h127.1 <dump.sql 

否则执行设置GTID_PURGED的SQL时会报下面的错误

ERROR 1840 (HY000) at line 24: @@GLOBAL.GTID_PURGED can only be set when @@GLOBAL.GTID_EXECUTED is empty. 

此时恢复出的MySQL实例的GTID_EXECUTED和备份时点一致

mysql> show master status;+---------------+----------+--------------+------------------+-------------------------------------------+| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                         |+---------------+----------+--------------+------------------+-------------------------------------------+| binlog.000002 |      191 |              |                  | e10c75be-5c1b-11e6-ab7c-000c296078ae:1-10 |+---------------+----------+--------------+------------------+-------------------------------------------+1 row in set (0.00 sec) 

由于恢复出的MySQL实例已经被设置的正确的GTID_EXECUTED,以master_auto_postion = 1的方式CHANGE MASTER到原来的主节点即可开始复制。

CHANGE MASTER TO MASTER_HOST='node1', MASTER_USER='repl', MASTER_PASSWORD='repl', MASTER_AUTO_POSITION = 1 

如果不希望备份文件中生成设置GTID_PURGED的SQL,可以给mysqldump传入--set-gtid-purged=OFF关闭。

MHA在切换时,如果使用非GTID模式切换,则在CHANGE MASTER中不会带上参数master_auto_position=0,而如果该从库之前配置为master_auto_position=1,那么CHANGE MASTER会报错,无法正常进行切换。

GTID与备份恢复

在做备份恢复的时候,有时需要恢复出来的MySQL实例可以作为Slave连上原来的主库继续复制,这就要求从备份恢复出来的MySQL实例拥有和数据一致的gtid_executed值。这也是通过设置gtid_purged实现的,下面看下mysqldump做备份的例子。

图片 1View Code

与MySQL传统复制相比,GTID有哪些独特的复制姿势?

##==========================================##
在非GTID模式下,会先进行Phase 3.1阶段,从拥有最新BINLOG的从库上获取差异日志,再进行Phase 3.2阶段,尝试从原Master服务器上获取最新BINLOG。
 图片 2

如何修复复制错误

在基于GTID的复制拓扑中,要想修复Slave的SQL线程错误,过去的SQL_SLAVE_SKIP_COUNTER方式不再适用。需要通过设置gtid_next或gtid_purged完成,当然前提是已经确保主从数据一致,仅仅需要跳过复制错误让复制继续下去。比如下面的场景:

在从库上创建表tb1

mysql> set sql_log_bin=0;Query OK, 0 rows affected (0.00 sec)mysql> create table tb1(id int primary key,c1 int);Query OK, 0 rows affected (1.06 sec)mysql> set sql_log_bin=1;Query OK, 0 rows affected (0.00 sec) 

在主库上创建表tb1

mysql> create table tb1(id int primary key,c1 int);Query OK, 0 rows affected (1.06 sec) 

由于从库上这个表已经存在,从库的复制SQL线程出错停止。

mysql> show slave statusG*************************** 1. row ***************************               Slave_IO_State: Waiting for master to send event                  Master_Host: 192.168.125.134                  Master_User: sn_repl                  Master_Port: 3306                Connect_Retry: 60              Master_Log_File: binlog.000001          Read_Master_Log_Pos: 1422               Relay_Log_File: mysqld-relay-bin.000003                Relay_Log_Pos: 563        Relay_Master_Log_File: binlog.000001             Slave_IO_Running: Yes            Slave_SQL_Running: No              Replicate_Do_DB:           Replicate_Ignore_DB:            Replicate_Do_Table:        Replicate_Ignore_Table:       Replicate_Wild_Do_Table:   Replicate_Wild_Ignore_Table:                    Last_Errno: 1050                   Last_Error: Error 'Table 'tb1' already exists' on query. Default database: 'test'. Query: 'create table tb1(id int primary key,c1 int)'                 Skip_Counter: 0          Exec_Master_Log_Pos: 1257              Relay_Log_Space: 933              Until_Condition: None               Until_Log_File:                 Until_Log_Pos: 0           Master_SSL_Allowed: No           Master_SSL_CA_File:            Master_SSL_CA_Path:               Master_SSL_Cert:             Master_SSL_Cipher:                Master_SSL_Key:         Seconds_Behind_Master: NULLMaster_SSL_Verify_Server_Cert: No                Last_IO_Errno: 0                Last_IO_Error:                Last_SQL_Errno: 1050               Last_SQL_Error: Error 'Table 'tb1' already exists' on query. Default database: 'test'. Query: 'create table tb1(id int primary key,c1 int)'  Replicate_Ignore_Server_Ids:              Master_Server_Id: 1                  Master_UUID: e10c75be-5c1b-11e6-ab7c-000c296078ae             Master_Info_File: mysql.slave_master_info                    SQL_Delay: 0          SQL_Remaining_Delay: NULL      Slave_SQL_Running_State:            Master_Retry_Count: 86400                  Master_Bind:       Last_IO_Error_Timestamp:      Last_SQL_Error_Timestamp: 161203 15:14:17               Master_SSL_Crl:            Master_SSL_Crlpath:            Retrieved_Gtid_Set: e10c75be-5c1b-11e6-ab7c-000c296078ae:5-6            Executed_Gtid_Set: e10c75be-5c1b-11e6-ab7c-000c296078ae:1-5                Auto_Position: 11 row in set (0.00 sec) 

从上面的输出可以知道,从库已经执行过的事务是'e10c75be-5c1b-11e6-ab7c-000c296078ae:1-5',执行出错的事务是'e10c75be-5c1b-11e6-ab7c-000c296078ae:6',当前主备的数据其实是一致的,可以通过设置gtid_next跳过这个出错的事务。

在从库上执行以下SQL:

mysql> set gtid_next='e10c75be-5c1b-11e6-ab7c-000c296078ae:6';Query OK, 0 rows affected (0.00 sec)mysql> begin;Query OK, 0 rows affected (0.00 sec)mysql> commit;Query OK, 0 rows affected (0.00 sec)mysql> set gtid_next='AUTOMATIC';Query OK, 0 rows affected (0.00 sec)mysql> start slave;Query OK, 0 rows affected (0.02 sec) 

设置gtid_next的方法一次只能跳过一个事务,要批量的跳过事务可以通过设置gtid_purged完成。假设下面的场景:

主库上已执行的事务

mysql> show master status;+---------------+----------+--------------+------------------+-------------------------------------------+| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                         |+---------------+----------+--------------+------------------+-------------------------------------------+| binlog.000001 |     2364 |              |                  | e10c75be-5c1b-11e6-ab7c-000c296078ae:1-10 |+---------------+----------+--------------+------------------+-------------------------------------------+1 row in set (0.00 sec) 

从库上已执行的事务

mysql> show master status;+---------------+----------+--------------+------------------+------------------------------------------+| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                        |+---------------+----------+--------------+------------------+------------------------------------------+| binlog.000001 |     1478 |              |                  | e10c75be-5c1b-11e6-ab7c-000c296078ae:1-6 |+---------------+----------+--------------+------------------+------------------------------------------+1 row in set (0.00 sec) 

假设经过修复从库已经和主库的数据一致了,但由于复制错误Slave的SQL线程依然处于停止状态。现在可以通过把从库的gtid_purged设置为和主库的gtid_executed一样跳过不一致的GTID使复制继续下去,步骤如下。

在从库上执行

mysql> reset master;Query OK, 0 rows affected (0.01 sec)mysql> set GLOBAL gtid_purged='e10c75be-5c1b-11e6-ab7c-000c296078ae:1-10';Query OK, 0 rows affected (0.03 sec)mysql> show master status;+---------------+----------+--------------+------------------+-------------------------------------------+| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                         |+---------------+----------+--------------+------------------+-------------------------------------------+| binlog.000002 |      191 |              |                  | e10c75be-5c1b-11e6-ab7c-000c296078ae:1-10 |+---------------+----------+--------------+------------------+-------------------------------------------+1 row in set (0.00 sec) 

此时从库的Executed_Gtid_Set已经包含了主库上'1-10'的事务,再开启复制会从后面的事务开始执行,就不会出错了。

mysql> start slave;Query OK, 0 rows affected (0.01 sec) 

使用gtid_next和gtid_purged修复复制错误的前提是,跳过那些事务后仍可以确保主备数据一致。如果做不到,就要考虑pt-table-sync或者拉备份的方式了。

如果群集因为某种原因导致主从节点上的Executed_Gtid_Set不同,如:

如何产生GTID

GTID的生成受gtid_next控制。 在Master上,gtid_next是默认的AUTOMATIC,即在每次事务提交时自动生成新的GTID。它从当前已执行的GTID集合(即gtid_executed)中,找一个大于0的未使用的最小值作为下个事务GTID。同时在binlog的实际的更新事务事件前面插入一条set gtid_next事件。

以下是一条insert语句生成的binlog记录

mysql> use `test`Database changedmysql> insert into tbx1 values(1);Query OK, 1 row affected (0.01 sec)mysql> show binlog events IN 'binlog.000015';+---------------+-----+----------------+-----------+-------------+-------------------------------------------------------------------+| Log_name      | Pos | Event_type     | Server_id | End_log_pos | Info                                                              |+---------------+-----+----------------+-----------+-------------+-------------------------------------------------------------------+...| binlog.000015 | 707 | Gtid           |         1 |         755 | SET @@SESSION.GTID_NEXT= 'e10c75be-5c1b-11e6-ab7c-000c296078ae:9' || binlog.000015 | 755 | Query          |         1 |         834 | BEGIN                                                             || binlog.000015 | 834 | Query          |         1 |         934 | use `test`; insert into tbx1 values(1)                            || binlog.000015 | 934 | Xid            |         1 |         965 | COMMIT /* xid=20 */                                               | 

在Slave上回放主库的binlog时,先执行set gtid_next ...,然后再执行真正的insert语句,确保在主和备上这条insert对应于相同的GTID。

一般情况下,GTID集合是连续的,但使用多线程复制(MTS)以及通过gtid_next进行人工干预时会导致gtid空洞。比如下面这样:

mysql> show master status;+---------------+----------+--------------+------------------+------------------------------------------+| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                        |+---------------+----------+--------------+------------------+------------------------------------------+| binlog.000015 |      965 |              |                  | e10c75be-5c1b-11e6-ab7c-000c296078ae:1-9 |+---------------+----------+--------------+------------------+------------------------------------------+1 row in set (0.00 sec)mysql> set gtid_next='e10c75be-5c1b-11e6-ab7c-000c296078ae:12';Query OK, 0 rows affected (0.00 sec)mysql> begin;Query OK, 0 rows affected (0.00 sec)mysql> commit;Query OK, 0 rows affected (0.00 sec)mysql> set gtid_next='AUTOMATIC';Query OK, 0 rows affected (0.00 sec)mysql> show master status;+---------------+----------+--------------+------------------+---------------------------------------------+| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                           |+---------------+----------+--------------+------------------+---------------------------------------------+| binlog.000015 |     1158 |              |                  | e10c75be-5c1b-11e6-ab7c-000c296078ae:1-9:12 |+---------------+----------+--------------+------------------+---------------------------------------------+1 row in set (0.00 sec) 

继续执行事务,MySQL会分配一个最小的未使用GTID,也就是从出现空洞的地方分配GTID,最终会把空洞填上。

mysql> insert into tbx1 values(1);Query OK, 1 row affected (0.01 sec)mysql> show master status;+---------------+----------+--------------+------------------+----------------------------------------------+| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                            |+---------------+----------+--------------+------------------+----------------------------------------------+| binlog.000015 |     1416 |              |                  | e10c75be-5c1b-11e6-ab7c-000c296078ae:1-10:12 |+---------------+----------+--------------+------------------+----------------------------------------------+1 row in set (0.00 sec) 

这意味着严格来说我们即不能假设GTID集合是连续的,也不能假定GTID序号大的事务在GTID序号小的事务之后执行,事务的顺序应由事务记录在binlog中的先后顺序决定。

处理办法:

基于GTID的复制如何工作

在MASTER_AUTO_POSITION = 1的情况下 ,MySQL会使用COM_BINLOG_DUMP_GTID协议进行复制。过程如下:

备库发起复制连接时,将自己的已接受和已执行的gtids的并集(后面称为slave_gtid_executed)发送给主库。即下面的集合:

UNION(@@global.gtid_executed, Retrieved_gtid_set - last_received_GTID) 

主库将自己的gtid_executed与slave_gtid_executed的差集的binlog发送给Slave。主库的binlog dump过程如下:

  1. 检查slave_gtid_executed是否是主库gtid_executed的子集,如否那么主备数据可能不一致,报错。
  2. 检查主库的purged_executed是否是slave_gtid_executed的子集,如否代表缺失备库需要的binlog,报错
  3. 从最后一个Binlog开始扫描,获取文件头部的PREVIOUS_GTIDS_LOG_EVENT,如果它是slave_gtid_executed的子集,则这是需要发送给Slave的第一个binlog文件,否则继续向前扫描。
  4. 从第3步找到的binlog文件的开头读取binlog记录,判断binlog记录是否已被包含在slave_gtid_executed中,如果已包含跳过不发送。

从上面的过程可知,在指定MASTER_AUTO_POSITION = 1时,Master发送哪些binlog记录给Slave,取决于Slave的gtid_executed和Retrieved_Gtid_Set以及Master的gtid_executed,和relay_log_info以及master_log_info中保存的复制位点没有关系。

图片 3

本文为DBA+社群的投稿文章:http://dbaplus.cn/news-11-857-1.html


##==========================================##
在基于GTID模式下,不会进行Phase 3.2阶段,即尝试从原Master服务器中获取最新BINLOG。

基于binlog文件位置的复制

在基于binlog文件位置的复制下,要保证crash safe slave,配置下面的参数即可。

relay_log_info_repository      = TABLErelay_log_recovery             = ON 

这样可行的原因是,relay_log_info_repository = TABLE时,apply

2、群集做过版本升级,从未使用GTID的版本升级到GTID版本,从库上曾一段时间内作为主库提供服务,但该时间段日志被Purged掉

Master的crash safe

前面一直在讲crash safe slave,Master的crash safe同样重要。 要想Master保持crash safe需要按下面的参数进行设置,否则不仅会丢失事务,gtid_executed还可能和实际的innodb存储引擎中的数据不一致。

sync_binlog                    = 1innodb_flush_log_at_trx_commit = 1 

在Master配置为"双1"的情况下,Master crash后,如果没有发生failover,可以继续作为Master。 如果发生了failover,可以检查旧Master和新Master上由旧Master执行的事务集合是否一致。

 show master status 

如果一致,可以按MASTER_AUTO_POSITION = 1的方式将旧Master作为Slave和新Master建立复制关系。否则,考虑做事务补偿或从新Master上拉取备份进行恢复。

在Master配置不是"双1"的情况下,在Master crash后由于难以准确知道旧Master上究竟执行了哪些事务,安全的做法是实施主备切换,并从新Master上拉取备份,把旧Master作为新Master的Slave进行恢复。

http://www.bkjia.com/HTML5/1181452.htmlwww.bkjia.comtruehttp://www.bkjia.com/HTML5/1181452.htmlTechArticle与MySQL传统复制相比,GTID有哪些独特的复制姿势? 本文为DBA+社群的投稿文章:http://dbaplus.cn/news-11-857-1.html 与MySQL传统复制相比,GTID有哪些...

##==========================================##
基于GTID模式进行故障切换时,无论原Master节点OS是否正常,都不会尝试从原Master节点读取BINLOG进行日志补偿。
基于GTID模式的MHA支持在复制拓扑中使用BINLOG Server来进行日志补偿,而非GTID模式的MHA会忽略BINLOG Server。
建议在基于GTID模式的群集中,不使用MHA进行"手动主从切换",该操作可能会导致原主库上部分BINLOG丢失。

event和更新relay_log_info表的操作被包含在同一个事务里,innodb要么让它们同时生效,要么同时不生效,保证位点信息和已经应用的事务精确匹配。同时relay_log_recovery

ON时,会抛弃master_log_info中记录的复制位点,根据relay_log_info的执行位置重新从Master获取binlog,这就回避了由于未同步刷盘导致的binlog文件接受位置和实际不一致以及relay log文件被截断的问题。

在同时使用MTS(multi-threaded slave)时,为保证crash safe slave基于binlog文件位置的复制还需要设置sync_relay_log=1,因为MySQL在Crash恢复时必须先通过读取relay log补齐MTS导致的事务空洞。

MySQL 5.6版本引入GTID来解决主从切换时BINLOG位置点难定位的问题,MHA从0.56版本开始支持基于GTID的复制,在发生故障切换时判断群集是否能采用基于GTID的方式进行切换

通过Xtrabackup进行备份

相比mysqldump,Xtrabackup是效率更高并且被广泛使用的备份方式。使用Xtrabackup进行备份的举例如下。

通过Xtrabackup创一个全量备份(可以在Slave上创建备份,以避免对主库的性能冲击)

innobackupex --defaults-file=/etc/my.cnf --host=127.1 --user=root --password=mysql --no-timestamp --safe-slave-backup --slave-info /mysql/bak 

应用日志

innobackupex --apply-log /mysql/bak 

查看备份目录中的xtrabackup_binlog_info文件可以找到备份时已经执行过的gtids

[[email protected] ~]# cat /mysql/bak/xtrabackup_binlog_infomysql_bin.000001    191 e10c75be-5c1b-11e6-ab7c-000c296078ae:1-10 

由于备份时添加了”--slave-info”选项并且从Slave节点拉取的备份,所以会生成xtrabackup_slave_info文件,也可以从这个文件里查找建立复制的SQL语句。

[[email protected] ~]# cat /mysql/bak/xtrabackup_slave_infoSET GLOBAL gtid_purged='e10c75be-5c1b-11e6-ab7c-000c296078ae:1-10';CHANGE MASTER TO MASTER_AUTO_POSITION=1 

将备份文件传送到新的节点node3的/mysql/bak目录并恢复(如果直接把备份传输到数据目录了,这一步可以省略)。

[[email protected] ~]# innobackupex --defaults-file=/etc/my.cnf --copy-back /mysql/bak 

启动MySQL。

[[email protected] ~]# mysqld --defaults-file=/home/mysql/etc/my.cnf --skip-slave-start & 

如果是从Slave拉的备份,一定不能直接开启Slave复制,这时的gtid_executed是错误的。需要手动设置gtid_purged后再start slave

reset master;SET GLOBAL gtid_purged='e10c75be-5c1b-11e6-ab7c-000c296078ae:1-10';CHANGE MASTER TO MASTER_HOST='node1',MASTER_USER='repl',MASTER_PASSWORD='repl',MASTER_AUTO_POSITION=1;start slave; 
Sun Jul  8 23:35:21 2018 - [info] MHA::MasterMonitor version 0.56.
Sun Jul  8 23:35:21 2018 - [info] GTID failover mode = 1
Sun Jul  8 23:35:21 2018 - [info] Dead Servers:
Sun Jul  8 23:35:21 2018 - [info] Alive Servers:
Sun Jul  8 23:35:21 2018 - [info]   10.0.203.104(10.0.203.104:3358)
Sun Jul  8 23:35:21 2018 - [info]   10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:35:21 2018 - [info]   10.0.203.117(10.0.203.117:3358)
Sun Jul  8 23:35:21 2018 - [info] Alive Slaves:
Sun Jul  8 23:35:21 2018 - [info]   10.0.203.104(10.0.203.104:3358)  Version=5.7.19-log (oldest major version between slaves) log-bin:enabled
Sun Jul  8 23:35:21 2018 - [info]     GTID ON
Sun Jul  8 23:35:21 2018 - [info]     Replicating from 10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:35:21 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
Sun Jul  8 23:35:21 2018 - [info]   10.0.203.117(10.0.203.117:3358)  Version=5.7.19-log (oldest major version between slaves) log-bin:enabled
Sun Jul  8 23:35:21 2018 - [info]     GTID ON
Sun Jul  8 23:35:21 2018 - [info]     Replicating from 10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:35:21 2018 - [info] Current Alive Master: 10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:35:21 2018 - [info] Checking slave configurations..
Sun Jul  8 23:35:21 2018 - [info]  read_only=1 is not set on slave 10.0.203.104(10.0.203.104:3358).
Sun Jul  8 23:35:21 2018 - [info]  read_only=1 is not set on slave 10.0.203.117(10.0.203.117:3358).
Sun Jul  8 23:35:21 2018 - [info] Checking replication filtering settings..
Sun Jul  8 23:35:21 2018 - [info]  binlog_do_db= , binlog_ignore_db= 
Sun Jul  8 23:35:21 2018 - [info]  Replication filtering check ok.
Sun Jul  8 23:35:21 2018 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Sun Jul  8 23:35:21 2018 - [info] Checking SSH publickey authentication settings on the current master..
Sun Jul  8 23:35:22 2018 - [info] HealthCheck: SSH to 10.0.203.109 is reachable.
Sun Jul  8 23:35:22 2018 - [info] 
10.0.203.109(10.0.203.109:3358) (current master)
 +--10.0.203.104(10.0.203.104:3358)
 +--10.0.203.117(10.0.203.117:3358)

Sun Jul  8 23:35:22 2018 - [warning] master_ip_failover_script is not defined.
Sun Jul  8 23:35:22 2018 - [warning] shutdown_script is not defined.
Sun Jul  8 23:35:22 2018 - [info] Set master ping interval 1 seconds.
Sun Jul  8 23:35:22 2018 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Sun Jul  8 23:35:22 2018 - [info] Starting ping health check on 10.0.203.109(10.0.203.109:3358)..
Sun Jul  8 23:35:22 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Sun Jul  8 23:35:58 2018 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
Sun Jul  8 23:35:58 2018 - [info] Executing SSH check script: exit 0
Sun Jul  8 23:35:58 2018 - [info] HealthCheck: SSH to 10.0.203.109 is reachable.
Sun Jul  8 23:35:59 2018 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Sun Jul  8 23:35:59 2018 - [warning] Connection failed 2 time(s)..
Sun Jul  8 23:36:00 2018 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Sun Jul  8 23:36:00 2018 - [warning] Connection failed 3 time(s)..
Sun Jul  8 23:36:01 2018 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Sun Jul  8 23:36:01 2018 - [warning] Connection failed 4 time(s)..
Sun Jul  8 23:36:01 2018 - [warning] Master is not reachable from health checker!
Sun Jul  8 23:36:01 2018 - [warning] Master 10.0.203.109(10.0.203.109:3358) is not reachable!
Sun Jul  8 23:36:01 2018 - [warning] SSH is reachable.
Sun Jul  8 23:36:01 2018 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and trying to connect to all servers to check server status..
Sun Jul  8 23:36:01 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun Jul  8 23:36:01 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Sun Jul  8 23:36:01 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Sun Jul  8 23:36:01 2018 - [info] GTID failover mode = 1
Sun Jul  8 23:36:01 2018 - [info] Dead Servers:
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:36:01 2018 - [info] Alive Servers:
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.104(10.0.203.104:3358)
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.117(10.0.203.117:3358)
Sun Jul  8 23:36:01 2018 - [info] Alive Slaves:
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.104(10.0.203.104:3358)  Version=5.7.19-log (oldest major version between slaves) log-bin:enabled
Sun Jul  8 23:36:01 2018 - [info]     GTID ON
Sun Jul  8 23:36:01 2018 - [info]     Replicating from 10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:36:01 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.117(10.0.203.117:3358)  Version=5.7.19-log (oldest major version between slaves) log-bin:enabled
Sun Jul  8 23:36:01 2018 - [info]     GTID ON
Sun Jul  8 23:36:01 2018 - [info]     Replicating from 10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:36:01 2018 - [info] Checking slave configurations..
Sun Jul  8 23:36:01 2018 - [info]  read_only=1 is not set on slave 10.0.203.104(10.0.203.104:3358).
Sun Jul  8 23:36:01 2018 - [info]  read_only=1 is not set on slave 10.0.203.117(10.0.203.117:3358).
Sun Jul  8 23:36:01 2018 - [info] Checking replication filtering settings..
Sun Jul  8 23:36:01 2018 - [info]  Replication filtering check ok.
Sun Jul  8 23:36:01 2018 - [info] Master is down!
Sun Jul  8 23:36:01 2018 - [info] Terminating monitoring script.
Sun Jul  8 23:36:01 2018 - [info] Got exit code 20 (Master dead).
Sun Jul  8 23:36:01 2018 - [info] MHA::MasterFailover version 0.56.
Sun Jul  8 23:36:01 2018 - [info] Starting master failover.
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] * Phase 1: Configuration Check Phase..
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] GTID failover mode = 1
Sun Jul  8 23:36:01 2018 - [info] Dead Servers:
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:36:01 2018 - [info] Checking master reachability via MySQL(double check)...
Sun Jul  8 23:36:01 2018 - [info]  ok.
Sun Jul  8 23:36:01 2018 - [info] Alive Servers:
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.104(10.0.203.104:3358)
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.117(10.0.203.117:3358)
Sun Jul  8 23:36:01 2018 - [info] Alive Slaves:
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.104(10.0.203.104:3358)  Version=5.7.19-log (oldest major version between slaves) log-bin:enabled
Sun Jul  8 23:36:01 2018 - [info]     GTID ON
Sun Jul  8 23:36:01 2018 - [info]     Replicating from 10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:36:01 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.117(10.0.203.117:3358)  Version=5.7.19-log (oldest major version between slaves) log-bin:enabled
Sun Jul  8 23:36:01 2018 - [info]     GTID ON
Sun Jul  8 23:36:01 2018 - [info]     Replicating from 10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:36:01 2018 - [info] Starting GTID based failover.
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] ** Phase 1: Configuration Check Phase completed.
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] * Phase 2: Dead Master Shutdown Phase..
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] Forcing shutdown so that applications never connect to the current master..
Sun Jul  8 23:36:01 2018 - [warning] master_ip_failover_script is not set. Skipping invalidating dead master IP address.
Sun Jul  8 23:36:01 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Sun Jul  8 23:36:01 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] * Phase 3: Master Recovery Phase..
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] The latest binary log file/position on all slaves is mysql-bin.000008:6689
Sun Jul  8 23:36:01 2018 - [info] Retrieved Gtid Set: 541e0f07-8047-11e8-8434-0800270b00d2:49-69
Sun Jul  8 23:36:01 2018 - [info] Latest slaves (Slaves that received relay log files to the latest):
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.104(10.0.203.104:3358)  Version=5.7.19-log (oldest major version between slaves) log-bin:enabled
Sun Jul  8 23:36:01 2018 - [info]     GTID ON
Sun Jul  8 23:36:01 2018 - [info]     Replicating from 10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:36:01 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.117(10.0.203.117:3358)  Version=5.7.19-log (oldest major version between slaves) log-bin:enabled
Sun Jul  8 23:36:01 2018 - [info]     GTID ON
Sun Jul  8 23:36:01 2018 - [info]     Replicating from 10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:36:01 2018 - [info] The oldest binary log file/position on all slaves is mysql-bin.000008:6689
Sun Jul  8 23:36:01 2018 - [info] Retrieved Gtid Set: 541e0f07-8047-11e8-8434-0800270b00d2:49-69
Sun Jul  8 23:36:01 2018 - [info] Oldest slaves:
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.104(10.0.203.104:3358)  Version=5.7.19-log (oldest major version between slaves) log-bin:enabled
Sun Jul  8 23:36:01 2018 - [info]     GTID ON
Sun Jul  8 23:36:01 2018 - [info]     Replicating from 10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:36:01 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.117(10.0.203.117:3358)  Version=5.7.19-log (oldest major version between slaves) log-bin:enabled
Sun Jul  8 23:36:01 2018 - [info]     GTID ON
Sun Jul  8 23:36:01 2018 - [info]     Replicating from 10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] * Phase 3.3: Determining New Master Phase..
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] Searching new master from slaves..
Sun Jul  8 23:36:01 2018 - [info]  Candidate masters from the configuration file:
Sun Jul  8 23:36:01 2018 - [info]   10.0.203.104(10.0.203.104:3358)  Version=5.7.19-log (oldest major version between slaves) log-bin:enabled
Sun Jul  8 23:36:01 2018 - [info]     GTID ON
Sun Jul  8 23:36:01 2018 - [info]     Replicating from 10.0.203.109(10.0.203.109:3358)
Sun Jul  8 23:36:01 2018 - [info]     Primary candidate for the new Master (candidate_master is set)
Sun Jul  8 23:36:01 2018 - [info]  Non-candidate masters:
Sun Jul  8 23:36:01 2018 - [info]  Searching from candidate_master slaves which have received the latest relay log events..
Sun Jul  8 23:36:01 2018 - [info] New master is 10.0.203.104(10.0.203.104:3358)
Sun Jul  8 23:36:01 2018 - [info] Starting master failover..
Sun Jul  8 23:36:01 2018 - [info] 
From:
10.0.203.109(10.0.203.109:3358) (current master)
 +--10.0.203.104(10.0.203.104:3358)
 +--10.0.203.117(10.0.203.117:3358)

To:
10.0.203.104(10.0.203.104:3358) (new master)
 +--10.0.203.117(10.0.203.117:3358)
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] * Phase 3.3: New Master Recovery Phase..
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info]  Waiting all logs to be applied.. 
Sun Jul  8 23:36:01 2018 - [info]   done.
Sun Jul  8 23:36:01 2018 - [info] Getting new master's binlog name and position..
Sun Jul  8 23:36:01 2018 - [info]  mysql-bin.000006:77499
Sun Jul  8 23:36:01 2018 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.0.203.104', MASTER_PORT=3358, MASTER_AUTO_POSITION=1, MASTER_USER='replicater', MASTER_PASSWORD='xxx';
Sun Jul  8 23:36:01 2018 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000006, 77499, 41d8a420-8047-11e8-8580-080027e837eb:1-92,
541e0f07-8047-11e8-8434-0800270b00d2:1-69
Sun Jul  8 23:36:01 2018 - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address.
Sun Jul  8 23:36:01 2018 - [info] ** Finished master recovery successfully.
Sun Jul  8 23:36:01 2018 - [info] * Phase 3: Master Recovery Phase completed.
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] * Phase 4: Slaves Recovery Phase..
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] * Phase 4.1: Starting Slaves in parallel..
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] -- Slave recovery on host 10.0.203.117(10.0.203.117:3358) started, pid: 5680. Check tmp log /var/log/masterha/app1/10.0.203.117_3358_20180708233601.log if it takes time..
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] Log messages from 10.0.203.117 ...
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info]  Resetting slave 10.0.203.117(10.0.203.117:3358) and starting replication from the new master 10.0.203.104(10.0.203.104:3358)..
Sun Jul  8 23:36:01 2018 - [info]  Executed CHANGE MASTER.
Sun Jul  8 23:36:01 2018 - [info]  Slave started.
Sun Jul  8 23:36:01 2018 - [info]  gtid_wait(41d8a420-8047-11e8-8580-080027e837eb:1-92,
541e0f07-8047-11e8-8434-0800270b00d2:1-69) completed on 10.0.203.117(10.0.203.117:3358). Executed 0 events.
Sun Jul  8 23:36:01 2018 - [info] End of log messages from 10.0.203.117.
Sun Jul  8 23:36:01 2018 - [info] -- Slave on host 10.0.203.117(10.0.203.117:3358) started.
Sun Jul  8 23:36:01 2018 - [info] All new slave servers recovered successfully.
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] * Phase 5: New master cleanup phase..
Sun Jul  8 23:36:01 2018 - [info] 
Sun Jul  8 23:36:01 2018 - [info] Resetting slave info on the new master..
Sun Jul  8 23:36:01 2018 - [info]  10.0.203.104: Resetting slave info succeeded.
Sun Jul  8 23:36:01 2018 - [info] Master failover to 10.0.203.104(10.0.203.104:3358) completed successfully.
Sun Jul  8 23:36:01 2018 - [info] 

----- Failover Report -----

app1: MySQL Master failover 10.0.203.109(10.0.203.109:3358) to 10.0.203.104(10.0.203.104:3358) succeeded

Master 10.0.203.109(10.0.203.109:3358) is down!

Check MHA Manager logs at localhost.localdomain:/var/log/masterha/app1/manager.log for details.

Started automated(non-interactive) failover.
Selected 10.0.203.104(10.0.203.104:3358) as a new master.
10.0.203.104(10.0.203.104:3358): OK: Applying all logs succeeded.
10.0.203.117(10.0.203.117:3358): OK: Slave started, replicating from 10.0.203.104(10.0.203.104:3358)
10.0.203.104(10.0.203.104:3358): Resetting slave info succeeded.
Master failover to 10.0.203.104(10.0.203.104:3358) completed successfully.

设置"双1"对性能的影响

出于安全考虑,强烈推荐设置"双1"。"双1"会增大每个事务的RT,但得益于MySQL的组提交机制,高并发下"双1"对系统整体tps的影响在可接受范围内。

sysbencholtp.lua10张表每张表100w记录(qps/并发数)图片 4

图片 5
对更新同一行这样无法有效并行的场景,"双1"对性能的影响非常大。

sysbenchupdate_non_index.lua1张表1条记录(qps/并发数)图片 6

图片 7
对不能有效并行的Slave replay,存在同样的问题。

通过指定tx-rate执行sysbench的update_non_index.lua脚本压测30秒,完成后检查主备延迟。

可以发现在Slave被配置为"双1"的情况下,延迟非常严重,1000以上的qps就会出现延迟,非"双1"下qps到5000以上才会出现延迟(主库配置为"双1")。

sysbenchupdate_non_index.lua1张表100w条记录 128并发(延迟/qps)图片 8

图片 9
以上测试环境是Percona Server 5.6运行在配置HDD的8 core虚机,由于测试结果和系统IO能力有很大关系,仅供参考。

##==========================================##
基于GTID进行故障切换:
1、如果候选Master节点不拥有最新的Relay log,那么将候选Master连接到拥有最新Relay log的Salve上进行日志补偿
2、如果群集中使用Binlog Server,则尝试从Binlog Server上拉取缺失的Binlog并应用到候选Master上
3、候选Matser拥有最新数据,将其升级为新Master,将其他slave连接到新Master上进行数据同步,可以给masterha_master_switch传入–wait_until_gtid_in_sync=1参数使其不等其它Slave完成数据同步,以加快切换速度。

GTID与crash safe slave

crash safe slave是MySQL 5.6提供的功能,意思是说在slave crash后,把slave重新拉起来可以继续从Master进行复制,不会出现复制错误也不会出现数据不一致。

因此不能简单修改Server.PM或DBHelper.pm文件来将基于GTID模式切换的群集修改为使用非GTID模式进行切换。

基于GTID的复制

上面的设置并不适用于基于GTID的复制。在基于GTID的复制下,crash的Slave重启后,从binlog中解析的gtid_executed决定了要apply哪些binlog记录,所以binlog必须和innodb存储引擎的数据保持一致。要做到这一点,需要把sync_binlog和innodb_flush_log_at_trx_commit都设置为1,即所谓的"双1"。

另外mysql启动时,会从relay log文件中获取已接收的GTIDs并更新Retrieved_Gtid_Set。由于relay log文件可能不完整,所以需要抛弃已接收的relay log文件。因此relay_log_recovery = ON也是必须的。

这样,对于基于GTID的复制,保证crash safe slave的设置就是下面这样。

sync_binlog                    = 1innodb_flush_log_at_trx_commit = 1relay_log_recovery             = ON 

关于如何设置以确保crash safe slave,官方文档有明确记载,见17.3.2 Handling an Unexpected Halt of a Replication Slave。

但是其中关于GTID的记载中存在笔误,将relay_log_recovery=1写成了relay_log_recovery=0(#83711)。同时也没有提到必须设置"双1",但是"双1"是必要的,否则crash的Slave重启后,可能会重复应用binlog event也可能会遗漏应用binlog event(#70659)。其中遗漏应用binlog event的情况更可怕,因为Slave在不触发SQL错误的情况下就默默的和Master不一致了。

本文由金沙官网线上发布于数据库,转载请注明出处:MySQL--MHA与GTID

您可能还会对下面的文章感兴趣: