Troubleshooting MariaDB Memory Leaks
Having a database is like having a car on the road. When everything is working, it is easy to overlook what is happening under the hood. But when it breaks, it must be repaired immediately.
We recently had a customer with MariaDB 10.6.18 on Ubuntu 22.04.5 LTS who had a strange issue — memory consumption kept increasing despite constant load. Regardless of how much database configuration was optimized, the problem persisted. It was like a tire leak, where it gradually consumed resources and eventually led to performance problems.
That's when we decided to dig deeper. Systematic troubleshooting led us to the culprit: glibc malloc, the system's default memory allocator. By switching to Jemalloc, we not only resolved the memory leak but also improved database stability.
In this blog, we're going to take you through our process of diagnosing the problem, making the decision to switch to Jemalloc, and the incredible outcome we saw. If you're running MariaDB and experiencing strange memory spikes, this post could save you hours of misery. So let's get started.
Memory Allocation in MariaDB
A memory allocator is responsible for managing how an application requests, uses, and releases memory from the operating system. The default memory allocator in Linux systems is glibc malloc, which is efficient for general-purpose use but can lead to fragmentation and excessive memory retention in long-running applications like databases.
Common Memory Allocators in Linux
- glibc malloc (default) – Standard memory allocator in Linux distributions, optimized for general use.
- Jemalloc – Designed to reduce memory fragmentation, commonly used for databases and caching systems.
- TCMalloc – Google's optimized memory allocator, suitable for multithreaded applications.
.png)
Since databases perform frequent memory allocations and deallocations, inefficient memory management can cause increased memory usage over time, leading to performance issues.
Identifying the Problem: Unusual Memory Growth
During routine monitoring, we noticed that MariaDB’s resident memory usage kept increasing, even when there was no significant increase in workload.
.png)
Symptoms We Observed
- Gradual increase in memory usage without an apparent reason.
- OS-level monitoring showed MariaDB consuming a significant amount of memory, far beyond expectations.
- Restarting MariaDB temporarily resolved the issue, but the memory creep would start again.
Initial Investigation: Ruling Out Common Causes
Before jumping to conclusions, we ruled out common culprits:
Misconfigured Buffers or Caches?
- Examined innodb_buffer_pool_size, query_cache_size, and other memory-intensive variables.
- No misconfigurations were found.
Long-Running Queries or Connections?
- Checked for long-running transactions, but they were normal.
- Connection pooling was working fine.
Table Scans or Inefficient Index Usage?
- Analyzed slow query logs, but no major anomalies.
.png)
With these ruled out, we suspected an OS-level memory leak or inefficient memory handling by due to bug.
Digging Deeper into the Root Cause
During our research, we came across relevant bug reports and discussions indicating that MariaDB’s default memory allocator (glibc malloc) could be causing memory fragmentation issues.
Relevant Bug Reports & Discussions
From the logs we could see that MariaDB was using the system’s default memory allocator.
mydbops@127.0.0.1:>SELECT @@version_malloc_library;
--------------
SELECT @@version_malloc_library
--------------
+--------------------------+
| @@version_malloc_library |
+--------------------------+
| system |
+--------------------------+
1 row in set (0.000 sec)
The Solution: Switching to Jemalloc
To fix this, we switched to Jemalloc, a more efficient memory allocator known for reducing fragmentation.
Steps to Implement Jemalloc
1. Install Jemalloc
sudo apt install libjemalloc22. Modify MariaDB Service to Use Jemalloc
Edit the systemd service file:
sudo systemctl edit mariadb
Add the following:
[Service]
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2"3. Restart MariaDB
sudo systemctl daemon-reload
sudo systemctl restart mariadb
After rebooting the MariaDB services, we confirmed that MariaDB was now using Jemalloc (version 5.2.1), successfully replacing the default memory allocator.
mydbops@127.0.0.1:(none)>SELECT @@version_malloc_library;
--------------
SELECT @@version_malloc_library
--------------
+------------------------------------------------------------+
| @@version_malloc_library |
+------------------------------------------------------------+
| jemalloc 5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756 |
+------------------------------------------------------------+Results: Dramatic Memory Improvement
- Memory consumption reduced by ~60%, as observed in OS monitoring.
- Stable memory usage over time, with no signs of leakage.
- Better performance in high-concurrency workloads.
Recommendations for MySQL/MariaDB Deployments
Considering the substantial improvement observed, we recommend:
- Adopting Jemalloc as a default optimization strategy for all new MariaDB/MySQL installations to enhance memory efficiency and reduce fragmentation.
- Incorporating this practice into our OS-level best practices and ensuring it is consistently implemented across deployments.
This case highlights the importance of deep OS-level debugging when facing unexpected resource consumption issues. Switching to Jemalloc proved to be a simple yet powerful solution for improving MariaDB’s memory efficiency. If you’re noticing unexplained memory spikes or performance degradation in your MariaDB environment, it’s a good time to investigate your memory management. Solutions like Jemalloc can make a world of difference.
At Mydbops, we specialize in MariaDB consulting and managed services. From diagnosing complex issues like memory leaks to optimizing your entire database environment, our team is here to ensure your systems run at peak performance. Explore Our MariaDB Services.
.svg)
