What you really want to do is implement a decent paging solution. This will solve both your problems, you won't have slow binding times and you won't use lots of server memory by caching larges amounts of data.
Humans can't handle 5000 records at once, it's too much information. Opinions vary, but for grid data I think 25-50 rows per page is about right - for what the eye can read and comprehend. So, if you're only displaying this amount of records at any one time, you only need to fetch that amount from your database. As you click 'next page', your grid can make an ajax request and fetch the next set of data to display.
Here's a GridView example using Update Panels http://www.c-sharpcorner.com/uploadfile/prathore/gridview-paging-and-sorting-using-ajax/
], but I strongly suggest you spend a bit of time searching - that's quite a basic example.
Using this technique, we have grids that happily display thousands of records.