Overview
This article explains the steps to follow when the following error is observed in the syslog file:
Jul 16 22:09:45 RTR_NAME tp_hub: App: '872_ProjAAA' - 'submit_sm' on outside session caused 'rtrNoRuleMatching' (0x04000004). Sent SMPP error 0x0045: Submit_sm or data_sm failed [15 times]
Requirements
You'll need the following for this troubleshooting procedure:
- VPN access to the customer.
- Lab system to double-check the process.
Root Cause Analysis
The node is receiving multiple consecutive submit_sm with the same sequence number.
- The first submit_sm is accepted.
- Another submit_sm sent prior to the first one being acknowledged is rejected because the sequence numbers need to be unique.
In the graph below you can see more than one submit_sm, sharing the same sequence number. The first message is accepted, but the second is rejected, with the same value we can see in syslog (0x00000045).
Resolution
The alarm can be caused by different situations, as explained below.
Lack of Active AO Rule in RTR (AO-MT)
The alarm can be due to an AO-MT issue, so verify that the AO rule for the application exists and is active in all the RTRs.
-
Obtain from the customer the AOR rule that is supposed to handle traffic for the application.
-
Connect to the customer's system and check each RTR to determine if the AOR rule exists.
Example: If the AOR rule index is 102:
tp_walk aortgruletable | grep "\.102" -
If the AOR rule doesn't exist in an RTR node, the customer will need to configure/activate it.
Wrong smsPropMaxStoreRequestsPerSecond Value
If the AOR rule exists in all the RTR nodes, the next possible cause to investigate is that the rtrNoRuleMatching alarm may be caused by a low value for the smsPropMaxStoreRequestsPerSecond parameter. Follow the steps detailed below to resolve this issue:
-
In the master MGR, backup the
common_config.txt
file in/usr/TextPass/etc.
-
Increase the
smsPropMaxStoreRequestsPerSecond
parameter in one of the following ways:- Remove the line for the parameter
maxstorerequestspersecond
in thecommon_config.txt
file to set the default value (2147483647), OR - Increase the value for the parameter
maxstorerequestspersecond
in thecommon_config.txt
file.
- Remove the line for the parameter
-
Save the file.
-
Validate the new configuration:
tp_config --validatecommonconfig common_config.txt
-
Restart the textpass process in all the traffic element nodes — including all instances, if applicable — to apply changes. This will require a maintenance window since it will affect traffic.
-
After the restart, verify that the parameter is set to the correct value.
tp_walk smsPropMaxStoreRequestsPerSecond
smsPropMaxStoreRequestsPerSecond.0 = INTEGER: 2147483647
Lack of Active AO Rule in RTR (AO-AT)
If the rtrNoRuleMatching alarm is still appearing for the same application, a new trace needs to be analyzed again for this application by following the steps below:
-
Connect to the customer traffic node where the alarm is appearing.
-
Get the port, as a textpass user:
tp_session | grep <application short code>
Example:
$ tp_session | grep 8721
8721 localhost:0006ED9B IP 11.222.333.444:36561 localhost:55500 outside
8721 localhost:0006ED9C IP 11.222.333.444:60882 localhost:55500 outside - Run tcpdump to capture the application traffic.
tcpdump -i any -s 0 -w /<path>/<filename>.pcap port <port used by application>
Example:
tcpdump -i any -s 0 -w /var/TextPass/backup/872_ProjAAA.pcap port 55500 -
Run the trace for 30 seconds and press CTRL+C to stop.
-
In the trace collected, you may find that the application is sending to recipients that are not MSISDNs (e.g. 0, 111, 007).
Since the recipients are technically shortcodes, this is AO-AT routing and there should be a routing rule for this, which the customer can add. A lack of this rule will cause the message rejection with the alarm rtrNoRuleMatching.