Discover the powerful capabilities of MongoDB’s FTDC feature for real-time performance monitoring and troubleshooting. Learn how to enable, disable, and visualize diagnostic data, and modify default settings for optimal utilization.

1. FTDC Overview
2. How to Disable the FTDC?
3. FTDC default path
   1. Mongod
   2. Mongos
4. What Data Does FTDC Capture?
5. Additional System Metrics in FTDC
6. Modifying Default Directory and File Size for FTDC
   1. Modifying Default Directory Size
   2. Modifying Default Collection Size
7. Updating the diagnosticDataCollectionPeriodMillis Value
8. FTDC Use case
9. Privacy Protection Measures for diagnostic.data
10. How to visualise the diagnostic data

MongoDB’s FTDC (Full-Time Diagnostics Capture) is a robust diagnostic feature that captures detailed information about the performance, operations, and behavior of a MongoDB database. It provides valuable insights into the database’s functioning and aids in performance monitoring and troubleshooting.

In this blog post, we will delve into the powerful capabilities of MongoDB’s FTDC (Full-Time Diagnostic Data Capture) feature. We will explore how it captures comprehensive data about the performance and operation of a MongoDB instance, enabling administrators and developers to gain real-time visibility into their database’s health and performance.

FTDC Overview

FTDC (Full-Time Diagnostics Capture) is a powerful feature integrated into MongoDB that captures detailed information regarding the performance and operation of a MongoDB instance. It ensures continuous data capture, providing users with a real-time view of the database’s performance and operation.

The data captured by FTDC is stored in a compressed format that is not directly readable by end-users. This format ensures efficient storage utilization while maintaining data integrity.

Introduced in MongoDB 3.2 (via SERVER-19585), FTDC was specifically designed to incrementally collect diagnostic data from certain commands. This valuable information assists MongoDB support in troubleshooting and resolving issues effectively.

Among the notable features of MongoDB, FTDC stands out as one of the most significant ones. It captures data every second, enabling administrators and developers to monitor the database’s performance with high granularity. Additionally, to prevent excessive storage usage, FTDC imposes a maximum limit of 200MB for the diagnostic.data folder size.

How to Disable the FTDC?

To disable the FTDC (Full-Time Diagnostics Capture) feature in MongoDB and conserve disk space, reduce the storage overhead associated with continuously capturing and storing diagnostic data, you can follow the steps outlined below:

• Include the diagnosticDataCollectionEnabled: false option in the setParameter setting of your configuration file when starting up the mongod or mongos process.
• Open the MongoDB configuration file, typically located at /etc/mongod.conf or /usr/local/etc/mongod.conf.
• Add the following line to the configuration file:

 
setParameter:
  diagnosticDataCollectionEnabled: false
	

• Save the file and restart the MongoDB
• Alternatively, you can disable diagnostic data collection directly in the MongoDB shell using the command:

 
db.adminCommand({setParameter: 1, diagnosticDataCollectionEnabled: false})
	

By executing this command, the diagnosticDataCollectionEnabled parameter in the admin database is set to false. This disables the collection of diagnostic data in MongoDB, preventing it from gathering diagnostic information for telemetry purposes.

Disabling FTDC (Full-Time Diagnostic Data Capture) may lead to a higher demand for time and resources when analyzing and troubleshooting issues with assistance from MongoDB Inc. engineers. The FTDC data provides valuable insights into the server’s state and behavior, aiding in the quick and accurate identification of the root cause of problems.

FTDC serves as a valuable tool for diagnosing MongoDB deployment issues by capturing diagnostic data at regular intervals. It helps in detecting performance issues, resource bottlenecks, and other problems that may affect the MongoDB server.

If FTDC is disabled, diagnosing and resolving issues may become more challenging since crucial diagnostic data will not be available. Therefore, it is generally recommended to keep FTDC enabled to facilitate effective issue resolution.

FTDC default path

Mongod

To utilize MongoFTDC, you have the option to enable it either in the MongoDB configuration file or manually start it using the mongod executable. Once enabled, it will automatically capture data and save it in the designated location.

FTDC data files generated by mongod processes are stored in a directory called diagnostic.data within the storage.dbPath of the respective instances. All diagnostic data files are organized within this directory. For instance, if the dbPath is set as /data/db, the diagnostic data directory would be located at /data/db/diagnostic.data.

 
Default path for mongod 
 
/data/db/diagnostic.data
	

Mongos

Mongos processes store FTDC data files in a diagnostic directory relative to the systemLog.path log path setting. MongoDB truncates the logpath’s file extension and concatenates diagnostic.data to the remaining name. For example, if the path setting is specified as /var/log/mongodb/mongos.log, the diagnostic data directory would be located at /var/log/mongodb/mongos.diagnostic.data.

 
Default path for mongos  

var/log/mongodb/mongos.diagnostic.data.
	

What Data Does FTDC Capture?

FTDC captures data about MongoDB’s performance, such as database lock contention and resource utilization. It also captures data about MongoDB’s operation, such as replica set status and database events. The system.profile collection contains detailed information about each MongoDB operation, including the operation type, duration, and any errors that occurred during the operation.

To collect logs for FTDC, mongod or mongos can use the following commands:

Full-Time Diagnostic Data Capture (FTDC) was introduced in MongoDB 3.2 (via SERVER-19585). FTDC is a feature that captures the output of specific diagnostic commands and system statistics at regular intervals to help troubleshoot issues.

 
serverStatus: 
   db.serverStatus({tcmalloc: true})

replSetGetStatus:
   rs.status()

collStats for the local.oplog.rs collection (mongod only)

connPoolStats (mongos only)
	

• db.serverStatus({tcmalloc: true}): serverStatus is a MongoDB command used to retrieve various statistics and information about the current state of the MongoDB server instance. The {tcmalloc: true} parameter, when included, enables the tracking of memory allocation statistics using the tcmalloc library. This can provide useful information for identifying memory leaks and optimizing memory usage.
• rs.status(): Command that is used to retrieve the current status of a replica set. provides information about the current state of the replica set, including: The set name and the names and addresses of all members of the set, The current primary member and its state, The current election information for the set, including the last election time and the number of election attempts,The current configuration version of the set .
• collStats for the local.oplog.rs collection (mongod only): collStats is a MongoDB command used to retrieve statistics about a collection, including the size of the collection, the number of documents, the average document size, and the number of indexes. The local.oplog.rs collection is a special collection used by MongoDB for replication. It contains a record of all changes made to a replica set, allowing secondary members to catch up to the primary member’s current state.
• connPoolStats (mongos only): connPoolStats is a MongoDB command used to retrieve statistics about the connection pool used by a MongoDB shard router (mongos) instance. It provides information about the number of active and available connections, the maximum and minimum number of connections allowed, and other relevant information. This information can be used to monitor the health and performance of a MongoDB sharded cluster.

Additional System Metrics in FTDCFTDC is enabled, and its scope is broadened to include system metrics through SERVER-21818 (introduced in MongoDB 3.2.13) and SERVER-31400 (introduced in MongoDB 3.4.16). In MongoDB 3.2.13 and MongoDB 3.4.16, the FTDC (Full-Time Diagnostic Data Capture) feature was enhanced to include system metrics along with internal diagnostic commands. This enhancement was implemented through the SERVER-21818 and SERVER-31400 tickets in the MongoDB issue tracker.The system metrics collected by FTDC in MongoDB 3.2.13 and later versions include:

• Server configuration and startup parameters
• Server statistics, such as memory usage, CPU utilization, and network traffic.

 
CPU utilization (ex: /proc/stat)

Memory utilization (ex: /proc/meminfo)

Disk utilization related to performance (ex: /sys/block/*/stat)

Network performance statistics (/proc/net/netstat)
	

By collecting these system metrics, FTDC provides a more comprehensive view of the system’s performance and behavior. This can be very helpful in troubleshooting issues related to system resource utilization and performance.
Here’s a brief description of each of metrics:

• CPU utilization: Captured from the /proc/stat file on Linux-based systems, this metric includes information on the amount of time the CPU spends in various states (such as user mode, system mode, and idle mode) and can be used to analyze CPU usage and identify potential bottlenecks.
• Memory utilization: Captured from the /proc/meminfo file on Linux-based systems, this metric includes information on memory usage (such as total memory, free memory, and used memory) and can be used to analyze memory usage and identify potential memory leaks or other issues.
• Disk utilization related to performance: Captured from the /sys/block/*/stat file on Linux-based systems, this metric includes information on disk I/O operations (such as read/write operations and blocks transferred) and can be used to analyze disk usage and identify potential bottlenecks.
• Network performance statistics: Captured from the /proc/net/netstat file on Linux-based systems, this metric includes information on network traffic (such as the number of packets received and transmitted) and can be used to analyze network usage and identify potential network-related issues.

These examples highlight only a few of the system metrics that MongoDB can capture when FTDC is enabled and the scope is broadened. Depending on the host operating system and MongoDB version, there may be additional metrics captured as well

Modifying Default Directory and File Size for FTDC

Modifying Default Directory Size

The “diagnosticDataCollectionMaxDirectorySizeMB” setting determines the maximum size, in megabytes, allowed for the diagnostic.data directory. If the directory size exceeds this specified value, the system or application will automatically remove the oldest diagnostic files within the directory based on their timestamps. This mechanism ensures that the diagnostic data directory does not grow excessively and occupy excessive disk space.

The command, “mongod –setParameter diagnosticDataCollectionDirectorySizeMB=250,” sets the “diagnosticDataCollectionDirectorySizeMB” parameter to a value of 250 for the mongod process. This means that the maximum size of the diagnostic.data directory for the MongoDB instance will be restricted to 250 megabytes.

 
mongod --setParameter diagnosticDataCollectionDirectorySizeMB=250
	

The default value for the “diagnosticDataCollectionMaxDirectorySize” parameter is 200 megabytes (MB).

To modify the “diagnosticDataCollectionMaxDirectorySize” in the MongoDB shell, you can utilize the following command:

 
db.adminCommand({setParameter: 1, diagnosticDataCollectionDirectorySizeMB: 250})
	

Modifying Default Collection Size

The command, “mongod –setParameter diagnosticDataCollectionFileSizeMB=20,” sets the parameter “diagnosticDataCollectionFileSizeMB” to a value of 20 for the mongod process. This means that each individual diagnostic file within the diagnostic.data directory for the MongoDB instance will have a maximum size of 20 megabytes.

The default value for the “diagnosticDataCollectionFileSizeMB” parameter is 10 megabytes (MB).

 
mongod --setParameter diagnosticDataCollectionFileSizeMB=20
	

To change the “diagnosticDataCollectionMaxDirectorySize” parameter in the MongoDB shell, you can use the following command

 
db.adminCommand({ setParameter: 1, diagnosticDataCollectionFileSizeMB: 20 })
	

Updating the diagnosticDataCollectionPeriodMillis Value

To adjust the frequency at which FTDC (Full-Time Diagnostic Data Capture) collects diagnostic data in MongoDB, you can change the value of the “diagnosticDataCollectionPeriodMillis” parameter.

To modify this value, you can utilize the “setParameter” command in MongoDB. For instance, to set the interval to 5 seconds, you would execute the following command:

 
mongod --setParameter diagnosticDataCollectionPeriodMillis=5000
	

Besides setting the “diagnosticDataCollectionPeriodMillis” parameter through the MongoDB run command, you can also configure it within the MongoDB shell using the “db.adminCommand()” method. It sends an administrative command to the MongoDB server, instructing it to update the “diagnosticDataCollectionPeriodMillis” parameter to 5000 milliseconds.

 
db.adminCommand({setParameter: 1, diagnosticDataCollectionPeriodMillis: 5000})
	

FTDC Use case

MongoFTDC is specifically designed to operate continuously in the background, capturing data without causing any impact on the performance of the MongoDB database. This captured data is stored in a collection of files known as FTDC files, which play a crucial role in diagnosing issues, troubleshooting problems, and enhancing overall performance.

For administrators, FTDC data serves as a valuable resource for troubleshooting MongoDB performance issues. By analyzing this data, slow queries, lock contention, and other performance bottlenecks can be identified. Such insights enable administrators to optimize their MongoDB deployment and ensure smooth operations.

It’s important to note that FTDC data does not contain any sensitive or confidential information, such as specific queries, query results, or user credentials. Instead, it focuses on collecting diagnostic data related to the internal state and behavior of the MongoDB server. This includes database statistics, server metrics, and performance counters, which are crucial for troubleshooting and monitoring purposes carried out by MongoDB support teams and system administrators.

To ensure the security of sensitive information, FTDC is designed to capture only a subset of data that is considered safe for collection. Consequently, FTDC data does not include any user-specific data, such as the contents of user-created collections or indexes, nor does it involve user credentials or security certificates used by the system or MongoDB itself.

Privacy Protection Measures for diagnostic.data

It is important to highlight that the diagnostic.data collected by FTDC always excludes specific types of data to prioritize privacy. This includes excluding samples of queries, query predicates, query results, data from end-user collections or indexes, as well as system or MongoDB user credentials and security certificates. This exclusion applies regardless of the MongoDB version being used.

How to visualise the diagnostic data

To visualize the diagnostic data, we are utilizing the GitHub tool. You can clone the mongo-ftdc repository from GitHub to your local machine by executing the command “git clone https://github.com/simagix/mongo-ftdc.git“.

Upon running this command, Git will create a new directory called “mongo-ftdc” in your current working directory. It will then proceed to download all the files from the repository into that directory. You can navigate to the “mongo-ftdc” directory and utilize the files stored there for collecting and analyzing FTDC data from your MongoDB deployments.

Here’s a step-by-step guide:

Step 1: Launch the Docker Desktop application.

Step 2: Transfer the diagnostic data files to the folder located at “~/mongo-ftdc/diagnostic.data/“.

Step 3: Change the directory path to “~/mongo-ftdc“.

Step 4: Launch the Docker container for FTDC.

Step 5: Once the Docker container starts successfully, you can find the Grafana URL in the terminal, as illustrated in the provided screenshot.

Step 6 :  Open the URL in a web browser to view the generated graph for the diagnostic data file.

Step 7 : Once the analysis is complete, you can stop the FTDC Docker container by pressing “Ctrl + C” in the Terminal. This will gracefully halt the container and stop its execution.

If you want to completely remove the running containers, along with their associated networks, volumes, and images, you can execute the command “docker-compose down“. This command not only stops the containers but also cleans up all the resources created during the execution of the “docker-compose up” command. It ensures the thorough removal of all the resources related to the Docker Compose project, leaving your environment clean and tidy.

FTDC (Full-Time Diagnostic Data Capture) is a powerful feature in MongoDB that empowers administrators and developers with comprehensive insights into the performance and operation of their MongoDB deployment. It captures data continuously, offering a real-time view of MongoDB’s performance. By leveraging FTDC data, administrators can pinpoint performance bottlenecks, troubleshoot issues, and optimize their MongoDB deployment for seamless operation. FTDC is a valuable tool that enhances the management and efficiency of MongoDB instances.

We invite you to explore our website, where you will discover a plethora of informative blog posts covering a wide range of topics related to database consulting, support, and technology. If you found this particular blog post helpful, chances are you will find other articles that capture your interest and provide valuable insights and advice for your business or career.

If you require expert assistance in managing your database, do not hesitate to contact us today. Our team of highly skilled professionals is committed to delivering customized solutions that cater to your specific needs, ensuring the security, optimization, and accessibility of your data at all times. Allow us to assist you in unlocking the full potential of your data.