A fairly large RTOS application I'm working on uses a command pattern for communications between processes. That is, if process A needs to invoke a function in process B, then A creates a command and puts that into a queue. Then a separate command thread in process A executes that command which involves making a CORBA call to invoke the function in process B.
During debugging it has been observed that sometimes these command threads become deadlocked awaiting on the distant process to finish. I am looking for a good method of monitoring these command threads and at a minimum logging when they deadlock and perhaps even recovering from such a situation.
Currently, I have implemented a method where another monitor thread for process A is created and the two threads interact as such (pseudocode):
command_name = command.getName()
command_timeout = command.getTimeout()
print "Error: " + command_name + " timed out executing."
Is there a better way to implement thread monitoring?