Memory Leak in Sybase EAServer

Background

I recently posted about object life cycle in Sybase EAServer (Jaguar). EAServer provides object pools (cache) where objects are cached when a client lets go of it. Since, the object creation and destroy are so costly, having such an object pool helps with performance immensely.

The Issue

Last year, there was a memory leak in one of our EAServers. After debugging the issue, I narrowed it down to couple of methods. Whenever the methods were called, the memory usage seemed to increase. This kept happening until the server ran out of memory and crashed. During our peak season, our web users ended up calling this method more frequently, thus crashing the server sooner.

The real problem was that each time the user called these methods, a new instance of an object (nvo_rtn_processor) was created inside the function and was never let go.

When we looked at the code, the nvo_rtn_processor was being destroyed as expected, but still it was not released for some reason. After analyzing the issue, we found that the DESTROY actually caused the issue. Changed to call of_deactivate and it worked out.

A Little PB Background

When a new instance of a EA Server component is needed in PB, it’s done by calling CreateInstance method in TransactionServer object. This creates an instance on the EA Server side and passes back a reference to it in PB.

For example, to create an instance of nvo_rtn_processor, we issued:

l_transaction_service.CreateInstance(lnv_rtn_processor,"its_returns/nvo_rtn_processor")

Where lnv_rtn_processor is the local reference in PB.

When we are done with the object thus created, we need to let go of it, so it can be returned to pool or garbage collected (see my post on object life cycle). To do this, we could either DESTROY the object or deactivate it.

Typically DESTROY is used for objects that were created using CREATE statement. For objects that were created with CreateInstance, a better approach is to use SetComplete on the TransactionServer. (Unless object’s Automatic Deactivation flag is set, then PB will take care of this).

EAServer is a distributed environment where distributed components participate in distributed transactions. The components don’t commit/rollback themselves. They merely indicate that they can be committed or rolledback using SetComplete or SetAbort methods. When the Jaguar Transaction Processor sees that, it commits or aborts the component’s changes. This helps Transaction processor to commit or rollback all objects participating in a single distributed transaction.

Solving the Mystery

The original developer used DESTROY to get rid of nvo_rtn_processor. This should have worked. Our deactivate and destructor events for this object1 were almost identical. So the issue was not in whether it was destroyed or deactivated. The problem was actually that the component was never “Deactivated”. The key is that we are actually calling a function called of_deactivate which in turn calls SetComplete on the transaction Server. SetComplete actually did the trick – this released the component and thus returned to pool as expected.

GetContextService("TransactionServer", lts)
// SetComplete to allow instance pooling/destruction
li_result = lts.SetComplete() 

The key lesson is that DESTROY only destroys the handle (local variable) to the component, but not the component itself. When nvo_rtn_processor was destroyed, the PB reference and the container reference probably got destroyed, but the actual component was never let go by the TransactionServer object, as we never told it to, thus the memory leak. (If the automatic Demarcation/DeActivation was set for the component, we might not have seen this issue.)

Conclusion

For any component that is created through TransactionServer, we need to call SetComplete (or SetAbort in case of a failure) to complete transaction and deactivate the component. For a lot of components in our application, this is done automatically, by enabling the component’s Automatic Demarcation/DeActivation.

Note

The deactivate event for an object will be called when it is deactivated – after SetComplete is called in this case. Destructor event will be called when the object is being destroyed.

nvo_rtn_processor itself is not declared transactional, but the above discussion is still true irrespective of the transaction type of the component. Automatic Demarcation can be set for components that are not transactional as well. If this is not done, then we must use SetComplete/SetAbort to release the object explicitly.

Filed under: CodeProject, EAServer, Misc, Powerbuilder Tagged: easerver