When you use File I/O resources you must release them after using.
you can use Process Explorer http://technet.microsoft.com/en-us/sysinternals/bb896653
It has Find handle
functionality in Handle
menu. Go to find handle and enter file/folder name created with c# and it will find the application which is still aquiring handle of the files.
you can Right Click on the handle in Process explorer lower pane and click close handle
to release the handle manually.
finally make sure you are releasing resources such as file streams and disposing the objects for File IO you are using in code to create files/folders before trying to delete them.
Whenever you work with some file operating system assigns a handle and passes it to the application requesting the file. In your case your c# application/Visual Studio is having that handle. Now if any other application will try to delete/Edit the file you will get Access Denied warning. So you must release them. Programatically you should release the File IO objects/streams and to confirm that your c# application is keeping that handle you can use process explorer.
The code you are written is sufficient only but you need to make sure you are releasing resources for example ..
CurrentFileStream = null;
If you are still not sure you can post the exact code you are using.