Click here to Skip to main content
Click here to Skip to main content
Go to top

Out of memory issue in MyBatis

, 9 Mar 2014
Rate this:
Please Sign up or sign in to vote.
This tip is for solving out of memory issue for large result sets in MyBatis

Problem Statement

We have a java web service deployed on the WebSphere Application Server 7. Everything was fine till one day when the testing team observed OutOfMemory error in the log file. The web service was developed using Spring MyBatis. The initial code analysis did not show any leak suspects in the code. The OutOfMemory error was coming intermittently since its first occurrence.

Heap Analysis

After the initial code analysis it was more or less clear to us that the culprit for OutOfMemory errors could be spring or MyBatis. While search the internet for heap analyzer tools we came across IBM HeapAnalyzer. It is a nice tool to analyze heap dumps generated by Web Sphere Application Server. For more information on this tool as well as to download its jar please visit

https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=4544bafe-c7a2-455f-9d43-eb866ea60091

HeapAnalyzer helps in identifying the memory leak areas by analyzing the heap dumps. The ‘heap leak suspect’ feature shows you the top 3 to 5 java objects consuming maximum memory. This helps in identifying the part of code which would be causing the out of memory issue. WebSphere Application Server by default generates heap dump whenever an OutOfMemory error is encountered.

The Culprit

When we ran HeapAnalyzer on the heap dump generated for our applications we found that more than 90% of memory is occupied by an ArrayList inside MyBatis DefaultResultHandler. On doing further analysis we found the issue with the default implementation of DefaultResultHandler provided by MyBatis. The default implementation of ResultHandler creates an internal array to store the select query result before returning the result to invoker. In our case some select queries were returning more than 500k of big size records. Now when the DefaultResultHandler is trying to store all these in the ArrayList, the heap was getting exhausted and OutOfMemory error was thrown.

Solution Details

The solution we implemented to overcome the issue was to override the DefaultResultHandler. The idea was to divide the result of select query into small chunks. We override the DefaultResultHandler to load 50000 records in the internal ArrayList and then invoke a callback operation to process those records. When one set of records is processed the ArrayList is emptied and new set of records are stored in it. This way we can restrict the number of objects in memory at any given point of time and avoid OutOfMemory errors.

Below is the sample code for custom ResultHandler that does the job explained above.
public class CustomResultHandler implements ResultHandler {

    private ArrayList<HashMap<String,String>>
Here the CustomResultHandler implements the ResultHandler interface defined by MyBatis. Its handleResult method is invoked for each record returned from the database by select query.
We need to define a callback interface and implementation class used in the above code. A simple callback interface could be defined as below:
public interface CallbackInterface {

    public void processData(ArrayList<HashMap<String,String>> recordList);

}
And the corresponding implementation class could be defined as below:
public class CallbackImpl implements CallbackInterface {

    @Override
    public void processData(ArrayList<HashMap<String, String>> recordList) {
        System.out.println("Callback operation invoked for list size: " + recordList.size());

    }

}
Next we need to bind the custom result handler with the mapper. For this we have to use sqlSession.select() methods, to which we pass the result handler to be invoked for each record in the ResultSet returned by the select query.
public ArrayList<HashMap<String, String>>
Now when getAllUsers is invoked an instance of CustomResultHandler is passed. This enables the custom result handler be invoked by MyBatis instead of DefaultResultHandler. A simple way to invoke this service method is like below:
public static void main(String[] args) {
        
        ApplicationContext context = new ClassPathXmlApplicationContext("Beans.xml");
        
        UserService userService = (UserService) context.getBean("userService");
        
        CustomResultHandler objResultHandler = new CustomResultHandler();
        CallbackInterface objCallback = new CallbackImpl();
        objResultHandler.setCallbackClass(objCallback);
        
        System.out.println("Users List: " + userService.getAllUsers(objResultHandler));
        
        
         
}
If you configure and run the attached code you will see that processData of CallbackImpl is invoked after every 100 records. We can optimize the record set by changing the MAX_COUNT value in CustomResutlHandler.

References

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author


Comments and Discussions

 
-- There are no messages in this forum --
| Advertise | Privacy | Mobile
Web03 | 2.8.140916.1 | Last Updated 10 Mar 2014
Article Copyright 2014 by Davinder Singla
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid