How to Effectively Manage MongoDB Logs and System Resources
Have you ever felt like your MongoDB logs are as overwhelming as an overflowing inbox? Everything seems fine until, suddenly, your disk space runs out, and troubleshooting becomes a nightmare. Without proper log management, things can spiral out of control. But it doesn't have to be that way. A few simple practices — like automating log rotation, compressing old logs, and keeping only what’s necessary — can make all the difference.
Let’s dive in and get those logs under control.
Effective management of MongoDB log files and system resources is essential for ensuring database performance, stability and ease of troubleshooting. Below, we outline best practices and recommendations to streamline log file management and resource utilisation.
.png)
Enabling Proper Log Rotation
MongoDB logs grow quickly and can consume significant disk space if not rotated properly. Enabling log rotation ensures that logs are rotated periodically, preventing the log file from becoming too large and avoiding potential disk space issues. For example, periodic log rotation ensures that old logs do not accumulate and hinder database performance, especially in high-write environments.
.png)
Note: The logRotate command is not a replicated command. You must connect to each instance of a replica set and run logRotate to rotate the logs for replica set members.
Automating Log Rotation
While MongoDB provides built-in support for log rotation, you can also perform log rotation manually. For automated log rotation, consider writing a custom script to rotate the log file and compress older logs. Retaining at least 15 days of logs is recommended for troubleshooting and audit purposes. This duration strikes a balance between having enough historical data to investigate issues and minimising disk space usage. However, the optimal retention period may vary depending on specific use cases, such as regulatory requirements or the frequency of log reviews.
.png)
Manual Rotation for MongoDB Log and MongoDB Audit Log File
To rotate the log file manually, issue the logRotate command from the admin database in MongoDB:
use admin
db.adminCommand({ logRotate: 1 })This command triggers log rotation manually in MongoDB by renaming the current log file.
Additionally, you can rotate only the audit log and provide a custom message at the time of rotation: Reference Link: Audit Log.
db.adminCommand({ logRotate: "audit", comment: "Rotating audit log" })This command triggers Audit log rotation manually in MongoDB by renaming the current log file.
Example Script for Log Rotation for Mongo Log file
Create a script to:
- Rotate the current MongoDB log file.
- Compress the previous day’s log file.
- Remove logs older than 15 days.
Example configuration for logrotate:
vi /etc/logrotate.d/mongo
/var/log/mongodb/auditLog*.json {
daily # Rotate logs daily
rotate 10 # Keep last 10 logs
missingok # Ignore if log file is missing
compress # Compress old logs
delaycompress # Delay compression by one cycle to avoid issues
notifempty # Skip rotation if log file is empty
create 644 mongodb mongodb # Ensure correct ownership and permissions
sharedscripts # Ensures postrotate script runs only once
postrotate
if pgrep mongod >/dev/null; then
systemctl kill --signal=SIGUSR1 mongod 2>/dev/null || \
/bin/kill -SIGUSR1 `cat /var/run/mongodb/mongod.pid 2>/dev/null` >/dev/null 2>&1
fi
endscript
}
Notes on Configuration
- /var/log/mongodb/mongod*.log: Specify the exact MongoDB log file path. For example, if the log file is /log/mongodb/mongod.log, mention /log/mongodb/mongod*.log.
- create 644 mongodb mongodb: Adjust the user and group based on the MongoDB process. For instance, if MongoDB is running as mongod, specify create 644 mongod mongod.
- /bin/kill -SIGUSR1: Update the path to the mongod.pid file as specified in the MongoDB configuration file.
Example Script for Log Rotation for Audit Log file
Create a script to:
- Rotate the current Audit log file.
- Compress the previous day’s log file.
- Remove logs older than 15 days.
Example configuration for logrotate for Audit Log:
vi /etc/logrotate.d/mongoAudit
/var/log/mongodb/auditLog*.json
{
daily
missingok
compress
delaycompress
notifempty
sharedscripts
create 644 mongod mongod
rotate 10
postrotate
if pgrep mongod >/dev/null; then
systemctl kill --signal=SIGUSR1 mongod 2>/dev/null || \
/bin/kill -SIGUSR1 `cat /var/run/mongodb/mongod.pid 2>/dev/null` >/dev/null 2>&1
fi
endscript
}
Notes on Configuration
- /var/log/mongodb/auditLog*.json: Specify the exact MongoDB log file path. For example, if the log file is /log/mongodb/auditLog.json, mention /log/mongodb/auditLog*.json.
- create 644 mongod mongod: Adjust the user and group based on the MongoDB process. For instance, if MongoDB is running as mongodb, specify create 644 mongodb mongodb.
- /bin/kill -SIGUSR1: Update the path to the mongod.lock file as specified in the MongoDB configuration file. ( If the pid path is not present, you can use the mongod.lock file. This is particularly useful in Docker-based MongoDB services where mongod.pid may not be defined in the configuration file.)
Prerequisites for Log Management
- Separate Log Directory:
- Maintain the MongoDB log directory on a partition separate from the MongoDB data directory to avoid disk-related issues.
- Avoid Shared Partition:
- Storing log files on the same partition as the data can fill up available space, potentially leading to database operation failures due to insufficient disk space.
- Prevents Excessive Disk Usage:
- By separating the log directory, log files do not consume the disk space allocated for MongoDB data, ensuring smooth database operations.
- Reduces Disk IOPS Strain:
- Storing logs on the same partition as the data can increase disk IOPS usage, which could lead to unnecessary disk performance issues on the servers.
.png)
Recommended Configuration in mongod.conf
Clarify whether the following parameters are defaults or need to be explicitly set in your mongod.conf file to enable seamless log rotation:
vi /etc/mongod.conf
systemLog:
logAppend: true # Ensures logs are appended instead of being overwritten
logRotate: reopen # Allows external logrotate to manage MongoDB logs
Note: After modifying the above parameter in the mongo.conf file, it is necessary to restart the MongoDB service. (Any changes or additions to the mongod.conf file requires a restart of the MongoDB service to take effect.)
Explanation of Configuration Parameters
systemLog.logAppend
- Type: boolean
- Default: false
- Description: When set to true, MongoDB appends new entries to the existing log file when the server restarts. Without this setting, MongoDB backs up the existing log file and creates a new one.
systemLog.logRotate
- Type: string
- Default: rename
- Description: Determines the behavior of the logRotate command. Options include:
- rename: Renames the current log file.
- reopen: Closes and reopens the log file, following standard Linux/Unix log rotation practices. Use this option when utilizing the Linux/Unix logrotate utility to avoid log loss.
Note: If you specify reopen, you must also set systemLog.log Append to true.
Default Behaviors log rotate option in MongoDB
With logRotate: rename
This approach may create empty files with names ending in a date format:
-rw-r--r-- 1 mongod mongod 0 Dec 14 16:37 mongod.log.2024-12-14T16-37-14
-rw-r--r-- 1 mongod mongod 0 Dec 14 16:37 mongod.log.2024-12-14T16-37-28
-rw-r--r-- 1 mongod mongod 0 Dec 14 16:37 mongod.log.2024-12-14T16-37-40
-rw-r--r-- 1 mongod mongod 0 Dec 14 16:37 mongod.log.2024-12-14T16-37-57
With logRotate: reopen
A separate log rotation script is required for proper handling:
-rw-r--r-- 1 mongod mongod 1.2K Dec 14 16:39 mongod.log.2.gz
-rw-r--r-- 1 mongod mongod 3.0K Dec 14 16:39 mongod.log.1
-rw-r--r-- 1 mongod mongod 2.5K Dec 14 17:12 mongod.log
Key Takeaways
- Separate Log Directory: Always maintain logs on a separate partition to prevent disk issues in the MongoDB data directory.
- Enable Log Rotation: Use systemLog.logAppend and systemLog.logRotate settings in mongod.conf for seamless log rotation.
- Automate Log Management: Create scripts to compress and clean up old logs, retaining at least 15 days of data (Based on our use case we can maintain the Mongo Logs).
- Monitor Disk Usage: Regularly check disk space to ensure availability for both logs and data.
.png)
Struggling with MongoDB Log Management or Database Performance? At Mydbops, we offer expert MongoDB Managed Services, Consulting Solutions, and Remote DBA Support to help you streamline log management, optimize performance, and ensure database security. Contact Us Now to schedule a consultation and experience hassle-free database management with us.
.svg)
