Article Original Creation Date: 2014-03-06
Overview
Major delays are observed on the GUI and the number of informs maxes at 20K.
The executePolicies task showed errors during processing, it times out on a DB query after 601 seconds. Because of that the policies stop processing informs and because of that, the sprt_nc_inform_event table grows rapidly.
Thu Mar 06 02:23:08 EET 2014: Exception Thrown (java.lang.reflect.InvocationTargetException). Stack trace follows.
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.supportsoft.encore.system.timer.EncoreTimer.doCall(EncoreTimer.java:174)
at com.supportsoft.encore.system.timer.EncoreTimer.doLoop(EncoreTimer.java:115)
at com.supportsoft.encore.system.timer.EncoreTimer.main(EncoreTimer.java:198)
Caused by: java.rmi.RemoteException: Transaction Rolledback.; nested exception is:
weblogic.transaction.internal.TimedOutException: Transaction timed out after 600 seconds
Restarting the timer helps out 10 minutes, after that it may start again showing these errors and because of that big delays in the processing of other timer tasks too. In this version of ServiceGateway, these tasks run in a sequential way and because of that influence each others timing.
NOTE: Stopping the extra timer responsible for the purging table job results in a better performance of the executePolicies job, which narrows down the problem to temporary Database (DB) performance issues. (Row Lock contention).
Environment
Solaris 10
WebLogic 9,2
Oracle 10
SG 4.0.10
Resolution
Restart all related processes:
- stop apache on the front_ends
- restart WebLogic and the Timers on the WebLogic cluster
After the restart, everything processes fine again without delays and backlog.