restrict access to each file to a specific user

If you use windows authentication and impersonation, you can set file system level ACL's. But that's not a good idea.
You should make your own "file server" on asp.net base. The concept is here: File Download in ASP.NET and Tracking the Status of Success/Failure of Download[^]. You can extend it to check user access stored in a database for example, and of course to give access not only to a specified file.