First, what's telling you that you're service is useing 1.5GB of RAM?? If you're lookin in Task Manager, it's lying to you.
In simple terms, the .NET CLR RESERVES memory for your application, called the Managed Heap. Any objects you allocate in your app, like an array are allocated out of the Managed Heap. When you free that object, the memory is returned to the Heap, NOT to Windows! Since your application used a ton of memory in the past, the .NET CLR will try to hang on to that memory so that future allocations of any objects happen quickly.
But, if Windows needs the memory back, the .NET CLR is more than happy to return any unused Manage Heap memory back to Windows.
Task Manager looks at memory usage from Windows point of view, not from the .NET CLR.
So, if you want to see how much memory you app is REALLY using, you have to use PerfMon and the .NET Memory counters or a memory profiling tool, such as these
], or if you're using Visual Studio 2010, you can use the built in profilers.