Overview

Your PBC Database connection has got lost. You are experiencing some or all of the following symptoms:

• Charging requests not getting through, and failing
• Possible impact on Message delivery
• Errors on connection visible in the syslogs: pbcDbServerConnectionLost
• SNMP alarms in the external Fault Management Server (FMS)
• PBC not pulling Subscriber information (where applicable)

Solution

Prerequisites:

• SSH Access to Problematic system as textpass user
• Syslogs (/var/log/messages)
• Trace of the Charging request
• Message Logs (Either from LGV or decoded from Traffic element)

Solution Steps:

1. SSH into the problematic server as textpass user.
2. Analyze the syslogs for messages similar to following:
   TEKELEC_TEXTPASS .smsc-lab-rtr4 textpass pbc pbcDeviceDB ("PBC", "192.168.7.5", 0) A database server connection has been lost.
   Aug 21 12:17:10 smsc-lab-rtr4 snmptrapd[2436]: 12:17:10 TRAP6.TEXTPASS-PBC-MIB::pbcDbServerConnectionLost TEXTPASS-PBC-MIB::pbcDbServerHost.1 = STRING: "192.168.7.5" TEXTPASS-PBC-MIB::pbcDbServerPort.1 = Gauge32: 0 TEXTPASS-GEN-MIB::deviceType.0 = STRING: "PBC" from smsc-lab-rtr4
3. Check if the PBC service is running by executing the command tp_status
   
   
4. If the PBC service is down, execute the command tp_start --tp_pbc to start the PBC service, and check if the connectivity has been restored ()
   Caution! Possible Impact on Message delivery, so it's recommended that you do this during a Maintenance Window
5. Ping the External server IP with which the connectivity got lost.
6. If the connectivity is not through (ping fail):
   • Check if the reverse routes have been properly added or not. Add them if they are missing, and check connectivity again.
   • Check from the Network Firewall team if all Diameter (for external Diameter connectivity) or Mysql (3306) ports  are added in the Allowed List, and get them added if they are missing or misconfigured.
   • Check if the Far end Node (with which connectivity is down) is alive in the network. If it is an external non-Lithium node, please take this up with your Network team.
   • Check if the routes have been properly configured from your Network side.
   • It might also be a Network glitch, that sometimes causes a temporary fluctuation of links and gets recovered automatically.
   • Verify that the Charging interface (ethernet port ‘ethX’) is UP and available, by executing the command ip a and analyzing the output. It should be UP, as shown below:
     
     
     • If it is not UP, try to bring it up by executing the command /etc/sysconfig/network-scripts/ifup <name_of_interface> (as root)
     • Check for network service status systemctl status network, and give it a restart with systemctl restart network, as shown below
       Caution! This might cause service degradation, and should be done during an approved Maintenance Window
       
       
     • If it still does not come up, check for Hardware issues including Network Cards, ports and cables.
7. Once issue has been resolved, validate by 
   • Testing  IP reachability through ping -I <Charging-interface-name> <End_server_IP>, and/or
   • Looking for the ‘recover’ message for the issue in syslogs matching the time of resolution, as shown in snapshot below, and/or
     
     
   • Checking if ‘PBC DB Query Successful’ counters are increasing by executing the command tp_walk -P pbcdbtable | grep Query
     
     

Expected Outcome: 

Either the PBC connection to DB or external node would be restored