Overview
You are experiencing storage failure (rtrStorageFailure alarms) on your traffic nodes. The rtrStorageFailure alarms occur on multiple nodes or apps in the Lithium Cluster have a tendency to periodically stop, and then recur in the future at random.
The rtrStorageFailure log file displays the following failure or error message:
Sent SMPP error 0x0064: SMSC storage failure [8 times]
CBVAS2-RTR01 tp_hub: App: '1234_BCP' - 'submit_sm' on outside session caused 'rtrStorageFailure' (0x04000001). Sent SMPP error 0x0064: SMSC storage failure [12 times]
CBVAS2-RTR01 tp_hub: App: '1234_UREG' - 'submit_sm' on outside session caused 'rtrStorageFailure' (0x04000001). Sent SMPP error 0x0064: SMSC storage failure [3 times]
CBVAS2-RTR01 tp_hub: App: '1233_UREG' - 'submit_sm' on outside session caused 'rtrStorageFailure' (0x04000001). Sent SMPP error 0x0064: SMSC storage failure [2 times]
CBVAS2-RTR01 tp_hub: App: 'MB_SMSAPI' - 'submit_sm' on outside session caused 'rtrStorageFailure' (0x04000001). Sent SMPP error 0x0064: SMSC storage failure
CBVAS2-RTR01 tp_hub: App: '1234_ITS' - 'submit_sm' on outside session caused 'rtrStorageFailure' (0x04000001). Sent SMPP error 0x0064: SMSC storage failure [7 times]
CBVAS2-RTR01 tp_hub: App: '4567_DAA' - 'submit_sm' on outside session caused 'rtrStorageFailure' (0x04000001). Sent SMPP error 0x0064: SMSC storage failure
CBVAS2-RTR01 tp_hub: App: '8901_UREG' - 'submit_sm' on outside session caused 'rtrStorageFailure' (0x04000001). Sent SMPP error 0x0064: SMSC storage failure [5 times]
CBVAS2-RTR01 tp_hub: App: '2345_ITS' - 'submit_sm' on outside session caused 'rtrStorageFailure' (0x04000001). Sent SMPP error 0x0064: SMSC storage failure [3 times]
CBVAS2-RTR01 tp_hub: App: '5678_UREG' - 'submit_sm' on outside session caused 'rtrStorageFailure' (0x04000001). Sent SMPP error 0x0064: SMSC storage failure [6 times]
[textpass@CBVAS2-RTR01 log]$ more messages |grep rtrStorageFailure
Information
Perform the below troubleshooting steps to check whether the issue is occurring because of bulk traffic on certain apps.
1. Check the syslogs of the affected node(s) by running the following query:
[textpass@node1] cat /var/log/messages | egrep “rtrstorage | error | replica | congest”
In our specific case, we noted there were several instances of rtrStorageFailure, see below Screenshot. As per HUB operator Manual, this alarm is caused when Storing a message in the AMS fails.
2. Check with your Database Administrator (DBA) if any SMS applications are currently sending huge broadcasts of messages.
If this was not the case, further investigation may need to be performed.
Please raise a support request to the NewNet Support Team, but be sure to include the following information:
- Share the complete syslogs from all nodes
- Include information on whether this is the first time this problem has occurred.
- Include information on whether the issue is ongoing or has ceased.
<supportagent>Agents: Note that there is currently an 'In Progress' JIRA ticket related to this issue - ZTPS-25670</supportagent>