Galera replicates the data across the cluster in a virtually synchronous fashion. Based on the availability of cluster nodes the type of data transfer can vary. The important factor which decides the type of data transfer required here is the existing write set in gcache. Galera Cluster stores write-sets in a special cache called the GCache. GCache is a memory allocator for write-sets in XtraDB Cluster.

If missing write-sets required for joiner are available in the donor’s gcache then Incremental State Transfers (IST) are triggered where only the missing transactions are transferred to the joiner node.

Oppositely, if the missing write-sets required for the joiner are not available in the donor’s gcache then State Snapshot Transfers (SST) is triggered where a complete snapshot of the entire node is transferred.

Here, in this blog, we are going to have a very simple walk-through of prime SST methods used in the XtraDB / MariaDB cluster which can help you in debugging the issues during ongoing SST and can give you a detailed overview of the SST methods.

Basically, there are two different approaches where Galera Cluster transfers state from one node to another.

Logical:

This approach uses mysqldump as the SST method. This is one of the slowest and least used SST methods which blocks the writes on the donor for the duration of the data transfer.As this is not the recommended SST method to be used we are not going to discuss about that in this blog.

Physical:

This approach uses two prime methods used for SST i.e. rsync and xtrabackup-v2. These are the most used SST methods and are very fast as compared to the logical one.

• rsync
• rsync_wan
• xtrabackup-v2
• mariabackup
• clone

rsync:

It is one of the fastest State Snapshot Transfer methods. The rsync script runs on both donor and joining nodes in the form of client – server model. Joiner here acts as the server whereas the donor here acts as the client transferring the data to the joiner (server). The major disadvantage of using this method is that the donor becomes read-only during the SST period.

Unlike xtrabackup-v2 rsync does not require any credentials to transfer the data which reduces the configuration effort and avoids issues occurring in SST due to improper/insufficient privileges.

rsync_wan:

This method is similar to rsync but uses a delta transfer algorithm and transfers only the delta changes. The delta transfer algorithm is designed to minimize network traffic at the cost of possible extra IO. It is usually used if the cluster nodes are spread across WAN where network throughput is the bottleneck.

Xtrabackup-v2:

It is one of the widely used State Snapshot Transfer methods. The major advantage of using this method is its non-blocking method except for the very minimal time during syncing of system tables and Galera or binlog information. This method requires the wsrep_sst_auth variable to be set up with user and password similar across the cluster. This is the recommended option for XtraDB /MariaDB cluster 5.5.34 and above.

Mariabackup:

It is build on top of Xtrabackup and it is the compatible for MariaDB Cluster (10.2 & above ). MariaDB do not support Xtrabackup due to the core changes in innodb redo log , TDE and others.

Clone:

Since MySQL version, 8.0.22 new SST method as a clone is available for Galera Cluster for MySQL.It is based on the native MySQL clone plugin itself. It is the fastest sst_method but blocks the donor node for DDL execution during the SST.

wsrep_sst_method (Xtrabackup-V2)

innobackup.backup.log:

This is a file created on the donor host which consists of all the logs from the xtrabackup process which is executed to stream the data from donor to joiner. This file can be very useful in troubleshooting any type of SST issue.

innobackup.prepare.log:

This file is created on the joiner host which consists of logs related to the ‘prepare‘ phase. In this stage, Xtrabackup applies transaction logs to tablespaces (ibd files) copied from the donor. Errors that occurred during the apply log phase will be logged in this file.

innobackup.move.log :

This file is created on a joiner host which consists of logs of the ‘move‘ stage in which Xtrabackup moves files from the temporary (.sst) directory to the final location in the MySQL data directory.

wsrep_sst_method (rsync)

rsync_sst.conf:

This file is created on the joiner host only during the period when the rsync process is in progress. It consists of information about the joiner data directory along with the PID file path of the rsync process and log file at the joiner’s host. This file is removed from the joiner’s data directory once the rsync process completes.

rsync_sst.pid:

This file is created on the joiner host only during the period when the rsync process is in progress. It consists of the parent process id of the ongoing rsync process at the joiner node.

rsync_sst_log:

This file is created on the joiner host only during the period when the rsync process is in progress. It consists of logs of the ongoing rsync process.

rsync_sst_complete:

This file is created on the donor host and is persisted even when the rsync process completes. It consists of the values of the wsrep_local_state_uuid of the cluster and the sequence number of the last committed transaction (wsrep_last_committed) status variable.

Now we have an overview about the SST and its supporting files in XtraDB Cluster.

More Reads:

https://www.slideshare.net/MyDBOPS/state-transfer-with-galera

https://www.mydbops.com/blog/mariadbxtradb-cluster-default-disk-files-and-its-significance/