Overview
After a server is rebooted, you notice that some routes are missing. The server might also not be reachable from the O&M IP.
Solution
There are two possible causes for this issue:
If you find the issue is being caused by something else, check how to request support help.
Routes Were Not Made Persistent
The missing routes might have been added by a customer but not made persistent. Once the server was rebooted, they were lost. Ask the customer to add those routes again and make them persistent.
If there is a degradation, consider stopping that node (run tp_stop with the textpass user) so that the redundant nodes would take up all the traffic. Then, the customer may add the missing routes and make them persistent. Once the routes are added, start the services on this node (run tp_start as the textpass user).
Conflicts Caused by Backup Files
Another possibility, according to the Red Hat documentation (login required), is that backup files stored under /etc/sysconfig/ are causing conflicts. To verify if that is the case and fix the problem, please perform the following steps:
- Check the existing IP routes:
netstat -nr
- Move all the backup files from /etc/sysconfig/ to another folder.
- Edit the route files on the server to have all the IP routes follow the same syntax in the files. The name of the route files would be of the format /etc/sysconfig/network-scripts/route-<interface-name>. Eg: /etc/sysconfig/network-scripts/route-eth0
- Connect via iLO and restart network services:
systemctl network restart
- Check if all IP routes are correctly defined.
Requesting Help From Support
In case you need to contact the support team about this issue, please create a support ticket and include the following information:
- The output of tp_walkall from the node with the issue and a similar one which is working fine
- Semi-static files (/usr/TextPass/etc/common_config.txt and /usr/TextPass/etc/[hostname]_config.txt) from both nodes
- Check if there is a recently created file in /var/TKLC/core/. If there is, you will receive the steps to extract it.
- The output of tp_status as the textpass user
- The syslogs (/var/log/messages).
- If there is any network connectivity outage reported.
- The uptime output of MGR to check if MGR was also restarted.
- Were any routes recently added?
- The output of echo $TZ and date from MGR
- The output of m3ua_link.
- The output of ls -altrh /usr/TextPass/etc/MGR*
- The output of ls -altrh /usr/TextPass/etc/MGR* on the MGR node.
- The syslog (/var/log/messages) from MGR node.
- All STV logs from /var/log/STV/ from MGR node.
- Ping MGR from the troubled server and vice-versa and send the output.
The support team will analyze the details and come back to you.
Testing
To find out if the issue is solved, reboot the server and check that the IP routes are loaded correctly.