Error: "CMDB backup available space on partition is running low"

Overview

You may encounter errors related to low disk space for the CMDB backup similar to the following one:
ce-3 warning: CMDB backup available space on partition /mnt/vol63cmdb is running low

Such errors are caused by a large CMDB backup size on the cache engine, which causes the backup operation to fail.

Solution

The permanent solution for this issue is to upgrade PeerApp UltraBand to the latest version (this issue was resolved in the 5.7.18 version).

Method of Procedure (MOP) for upgrading Application,
1) Stop service 
2) Format storage 
2) Upgrade Application 
3) Reboot all CE's
5) Start service 

Note: In case you opt for an upgrade, kindly arrange a total 3 Hrs maintenance window for all these activities and update us in advance so we can check and align our resources for this activity.

A workaround can be obtained by clearing the cache content but it is recommended to get a system upgrade in order to resolve this and many other known issues. Refer to PeerApp Release Notes to find the latest available version of PeerApp and the issues that are resolved in each version.

The temporary solution is to remove the backup file(s) from the partition. This action has no impact on the services. Follow the process below to remove the backup files;

1. Connect via SSH to the Management IP.
2. Switch to superuser:
   su -
3. If it is a grid configuration, log into the cache engine related to the alarm.
   In this example, ce-3 is the corresponding cache engine.
   ssh ce-3
4. Retrieve the current CMDB status:
   for i in `df -kh | grep cmdb | awk '{print $6}'`; do echo $i; ls -lh $i; echo
   
   The output of this command should show the partition that consumes a lot of space:
   

   ===== done
   /mnt/vol63cmdb
   total 16G
   -rwx------ 1 root root 16G 2019-12-06 13:47 PA_stats.db
   =====
   /mnt/vol4cmdb
   total 0
   =====
   /mnt/vol6cmdb
   total 0
   =====
   /mnt/vol9cmdb
   total 0
   =====
   /mnt/vol16cmdb
   total 0
   =====
   /mnt/vol43cmdb
   total 0
   =====
   /mnt/vol57cmdb
   total 0
   =====
   /mnt/vol7cmdb
   total 0
   =====
   /mnt/vol55cmdb
   total 0
   =====
   -rwx------ 1 root root 3.2G 2019-10-24 16:02 PA_parts.db
   -rwx------ 1 root root 2.7G 2019-10-24 16:02 PA_stats.db
   =====

5. Remove the backup file from the affected partition. In this example, vol63cmdb is the affected partition. 
   rm -f /mnt/vol63cmdb/PA_stats.db
6. Check the CMDB status again:
   for i in `df -kh | grep cmdb | awk '{print $6}'`; do echo $i; ls -lh $i; echo
   
   The affected partition should not consume any space (total 0):

   ===== done
   /mnt/vol63cmdb
   total 0
   =====
   /mnt/vol4cmdb
   total 0
   =====
   /mnt/vol6cmdb
   total 0
   =====
   /mnt/vol9cmdb
   total 0
   =====
   /mnt/vol16cmdb
   total 0
   =====
   /mnt/vol43cmdb
   total 0
   =====
   /mnt/vol57cmdb
   total 0
   =====
   /mnt/vol7cmdb
   total 0
   =====
   /mnt/vol55cmdb
   total 0
   -rwx------ 1 root root 3.2G 2019-10-24 16:02 PA_parts.db
   -rwx------ 1 root root 2.7G 2019-10-24 16:02 PA_stats.db
   =====

Back to top