The Lounge - CodeProject

jarvisa13-Jul-20 0:21

13-Jul-20 0:21

I worked on a system that had to stream 1MB images to disk at 75fps. I found that once there were about 700 files in a directory, creating new files suddenly became slower and the required transfer rate was unachievable. I ended up creating a new subdirectory every 500 files.
Of course this won't be a problem if your system is purely for archive.

Re: Storing huge numbers of files

agolddog13-Jul-20 4:40

agolddog

13-Jul-20 4:40

I don't know about access issues for a large number of files in a directory, but you might also consider security issues.

If, for example, you have several different users whose files should not be accessible by the others, creating a subfolder for each user might allow you to secure them such that only their user has access to their subfolder (plus maybe some 'admin' user that you use which can see all directories). Obvious organizational advantages as well.

Re: Storing huge numbers of files

Member 123113713-Jul-20 7:43

Member 1231137

13-Jul-20 7:43

It really depends on your use case for accessing/managing these files. If you're going to be enumerating the files a lot (or portions of the files) then everything in one directory/folder may not be the best. You can at least "chunk up" the enumeration by subfolder if you create those.

Also, if you break them up into subfolders in some logical way, then managing those units and/or groupings of files will become much easier. I.E. Backups, restoring, archiving, deleting.

If you are storing the path to each file in a database, then you're going to get the same performance either way (subdirectories and everyone in the pool together).

Can you explain a little more about the repository and how you'll be using it?

Re: Storing huge numbers of files

englebart13-Jul-20 7:44

englebart

13-Jul-20 7:44

Consider drive corruption, backups, replication, file listeners, aging/document retention and all of the other access aspects as well. Folder per day/month/year can help out with some of those items as suggested on another post.

Re: Storing huge numbers of files

ddrogahn13-Jul-20 9:58

ddrogahn

13-Jul-20 9:58

You might get by if using SSDs, or files are large and accessed directly & infrequently, and won't increase by orders of magnitude.
Better to spread them out.

Huge directories in NTFS:
* Accessing individual files is OK
* Adding/removing/listing/sorting gets slow (consider EnumerateFiles instead of GetFiles)
* Reading metadata (mod date) is slow (makes Explorer detail view slow)
* Network access is slower
* Defragging directories (with contig) helps some (also moving large dirs with robocopy /create)

Directories (and empty/tiny files) are stored in the MFT.
A massive number of MFT entries can be a problem.
The MFT starting size is set when (and only when) you format the disk (controlled by a registry key). It will expand if needed (but fragment), and will contract (if possible) when space is low.
Defragging MFT is possible but slow and difficult.
After a disk was full of files, or had the MFT filled by directories or tiny files -- it may be best to reformat.

How to segment depends on how sparse the file IDs will be.
About 4k entries is a good starting target.
If files have numeric IDs: Avoid bit shifts, for simplicity.
Group into 3 digits (base 10) = 1000 files + 1000 subdirs
or 3 hex chars 0xFFF = upto 4k files + 4k subdirs

eg.
000/0.dat - 999.dat
001/1000.dat - 1999.dat
999/999000.dat - 999999.dat
...
000/001/1000000.dat - 1000999.dat
001/001/1001000.dat - 1001999.dat
123/987/987123000.dat - 987123999.dat

Re: Storing huge numbers of files

Jim Knopf jr.7-Aug-20 5:20

Jim Knopf jr.

7-Aug-20 5:20

Explorer does two things. Read the entries and sort them. Looks like reading is linear and sorting too. So you get N^2 time behavior. This didn’t change for decades.
Some file systems allow accessing the files with a kind of pointer, avoiding the directory once you know the pointer. Nevertheless, adding and deleting files still has to touch the directory.
Looking up directory names has the same problem. So better construct a directory “tree”.
File size doesn’t matter for name lookups.

Best keyboard ever

Eddy Vluggen11-Jul-20 6:19

Eddy Vluggen

11-Jul-20 6:19

A while ago I dropped half a mug of coffee over my cheap USB keyboard and of course it died. Been given a Corsair K68 RGB, a mechanical keyboard (+ non spill mug). Sturdy keys, good for bulk typing. So, few days ago, the unthinkable happened - I dropped a full mug of coffee on the new keyboard.

..and it still works, without flaws. There's an anti-spill rubber between the keys and the internals, and there's a metal covering under that which protects the electronics. I'm convinced that nothing is idiot proof and I will find a weak spot, but for now, I'm pretty impressed. It's easily taken apart and cleaned and might actually last longer than a year.

YouTube showing rain on a keyboard[^]

So, what's your keyboard?

Bastard Programmer from Hell Suspicious | :suss:

If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.

Welcome to the Lounge