TiDB is an open-source, distributed SQL database designed for mission-critical applications, providing scalability and strong consistency. One of the key features that enhance its performance and efficiency is data compression. This blog explores the various types of compression available in TiDB, their characteristics, and how to configure them for optimal performance.

Supported Compression in TiDB

TiDB supports several compression algorithms, each with unique characteristics and use cases.

The primary compression algorithms available are:

LZ4

LZ4 is known for its high-speed compression and decompression rates, offering a good balance between speed and compression ratio. It is particularly well-suited for scenarios where speed is more critical than the compression ratio.

Characteristics:

• LZ4 offers fast compression and decompression speeds, making it suitable for applications that prioritize speed.
• It achieves a moderate compression ratio, balancing performance and efficiency.

Use Cases:

• This algorithm is ideal for real-time data processing scenarios where swift data handling is essential.
• Additionally, it is well-suited for applications that require low latency to ensure quick response times.

Zlib

Zlib provides a higher compression ratio compared to LZ4, but it comes with slower compression and decompression speeds. It is ideal for situations where storage savings are more important than speed.

Characteristics:

• Zlib offers a higher compression ratio, providing more efficient data storage.
• However, this comes with slower compression and decompression performance.

Use Cases:

• Zlib is ideal for storage space optimization, making it suitable for scenarios where reducing data size is essential.

Snappy

Snappy is designed for high-speed compression and decompression with a moderate compression ratio. It is often used in scenarios where performance is crucial, such as real-time data processing.

Characteristics:

• Snappy provides fast compression and decompression with a moderate compression ratio.

Use Cases:

• Snappy is well-suited for applications requiring fast data processing and minimal latency.

Zstd (zstandard) Compression in TiDB

Zstandard (zstd) is a compression algorithm supported by TiDB that offers a good balance between compression ratio and decompression speed. It can be a compelling choice for various use cases in TiDB deployments.

Characteristics:

• Compression Ratio: Zstd generally achieves a higher compression ratio than LZ4 but lower than Zlib. This translates to more compact data storage on your TiKV nodes.
• Decompression Speed: Zstd offers faster decompression speeds compared to Zlib, making data retrieval more efficient.
• Balance: Zstd strikes a good middle ground between the high speed of LZ4 and the high compression ratio of Zlib.

Compression Ratio Comparison

To understand the differences between these algorithms, let's compare their compression ratios and performance characteristics:

For more insights on performance and security, check out our blog What's New in TiDB 8.1.0, the latest LTS release? Performance, Security, & More!

Configuring Compression in TiDB

You can set the compression algorithm for TiKV (the storage engine of TiDB) in the configuration file. Here's an example configuration for using LZ4 compression:

To configure compression in TiKV, you need to use the rocksdb section in the TiKV configuration file. Here is the correct format for setting compression using LZ4 in TiKV:

 
# tikv.yml
rocksdb:  
defaultcf:    
compression-per-level: ["no", "lz4", "lz4", "lz4", "lz4", "zstd", "zstd"]  
writecf:    
compression-per-level: ["no", "lz4", "lz4", "lz4", "lz4", "Zstd", "Zstd"]  
lockcf:    
compression-per-level: ["no", "no", "no", "no", "no", "no", "no"]  
raftcf:    
compression-per-level: ["no", "lz4", "lz4", "lz4", "lz4", "lz4", "lz4"]
	

Example

The example compression-per-level setting:

compression-per-level = ["no", "no", "lz4", "lz4", "lz4", "zstd", "zstd"]

Levels 0-1 have no compression.

Levels 2-4 use LZ4 compression, which provides a balance between speed and storage savings.

Levels 5-6 use Zstd compression, which offers higher compression ratios at a moderate speed.

In this Configuration,

• The compression-per-level setting specifies the compression algorithm for each level in the RocksDB engine.
• LZ4 compression is set for levels 1 to 6 in all Column Families (defaultcf, writecf, lockcf, raftcf).
• You can replace "lz4" with other supported compression algorithms like "zlib" or "snappy" based on your specific requirements.
• Zstd compression is set for levels 5 and 6 to achieve higher compression ratios at those levels.

In this example, LZ4 is set as the compression algorithm for all Column Families (defaultcf, writecf, lockcf, raftcf). You can replace "lz4" with "zlib" or "snappy" based on your needs.

For more on how TiDB manages replication, read TiDB's Raft-based Replication.

Choosing the Right Compression Algorithm

Choosing the right compression algorithm depends on your specific requirements for speed, compression ratio, and storage optimization. Here’s a simple flowchart to help you decide:

• LZ4 and Snappy: Choose these if your priority is high-speed data processing and low latency. They are ideal for real-time applications where quick access to data is necessary.
• Zlib: Choose this if your priority is maximizing storage efficiency by achieving a higher compression ratio, and you can tolerate slower compression and decompression speeds.
• Zstd: Choose this if you need a balance between compression ratio and speed, providing good performance with moderately high compression ratios.

For advanced features in TiDB, explore TiDB's Co-Processor: Distributed SQL with a Boost.

Choosing the right compression algorithm in TiDB depends on your specific use case and requirements. Here are some considerations to help you decide on the appropriate compression algorithm:

Speed vs. Compression Ratio

• LZ4: Offers high-speed compression and decompression rates but may have lower compression ratios compared to other algorithms.
• Zlib: Provides a balance between compression ratio and speed, making it suitable for scenarios where both factors are important.
• Snappy: Optimized for speed and efficiency, making it ideal for scenarios where low latency and high throughput are crucial.
• Zstd: Strikes a balance between speed and compression ratio, offering better compression ratios than LZ4 and Snappy with reasonably high speeds.

Data Characteristics

• Consider the nature of your data (e.g., structured, unstructured, repetitive patterns) to determine which compression algorithm would be most effective in reducing data size while maintaining performance.

Use Case

• Choose the compression algorithm based on the specific use case. For example, if you prioritize fast data processing, LZ4 or Snappy may be more suitable. If you need a good balance between compression ratio and speed, Zstd could be a better choice.

Testing and Benchmarking

• It is recommended to test and benchmark different compression algorithms with sample data to evaluate their performance in your TiDB environment. This can help you determine which algorithm works best for your workload.

Dynamic Configuration

• TiDB allows you to dynamically configure compression algorithms, so you can experiment with different algorithms and adjust them based on the performance results.

By considering these factors and testing the compression algorithms in your TiDB environment, you can choose the right compression algorithm that aligns with your performance, speed, and efficiency requirements.

To delve deeper into TiDB's capabilities, check out these blogs:

• Understanding Flashback in TiDB
• Leverage TiDB's Automated TTL Feature for Efficient Data Management
• TiDB Scheduling: The Secret Weapon for Peak Database Performance

Compression is a powerful tool to optimize storage and improve performance in TiDB. By understanding the different compression algorithms available, you can choose the best one for your specific use case. LZ4 offers a good balance between speed and compression ratio, Zlib provides higher compression ratios, and Snappy ensures high-speed operations. Zstd provides a balanced option with moderately high compression ratios and good decompression speeds, making it a versatile choice for many applications.

Unlock the full potential of your TiDB deployment with our expert services. Contact Mydbops today for performance tuning and optimization. We offer TiDB consulting services tailored to your specific needs.

{{cta}}