How does windows keep track of reserved virtual memory?

Question

0.00/5 (No votes)

See more:

I'm working on a C project for which I'm going to use a custom allocation scheme. I would like to have several memory arenas which I can allow to grow as much as necessary to accommodate the needs of the user. I would like to explicitly place the arenas in my 64-bit address space so as to spread them out approximately evenly, minimizing the chance that one arena will ever grow large enough to collide with another and force its reallocation.

This scheme relies on being able to know that no stray allocations will be made between the current end of one arena and the start of the next. However, I will be using winAPI for my GUI, which will presumably allocate in my address space, and as far as I can see, I can't dictate to its allocators where they're allowed to do so. It seems that instead, I would have to specify everywhere it isn't allowed to allocate by calling VirtualAlloc with MEM_RESERVE, and since the wider the arenas are spaced, the better, I would prefer that to entail as much of my address space as I can get away with.

So...would making a reservation of such a large span cause any kind of pathological behavior? People have suggested that it would gunk up Windows' system for tracking virtual memory, but never with a concrete enough explanation of how to convince me to give up on the idea. I understand that for committed memory, the OS must enumerate each page since it needs to specify how the page is mapped to physical memory. In other words, the amount of work scales with the amount of memory committed. If memory that is merely reserved is treated the same way, then my large reservation would indeed stress the system. However, since reserving lacks the complication of mapping to physical memory, I can imagine that instead of enumerating each page, it only keeps track of the start and end of each reservation. In that case, the amount of work would scale with the number of reservations, and my giant reservation would be no worse than reserving a single page.

What I have tried:

Reading about the Windows virtual memory model.

Posted 28-Sep-19 3:27am

Alexander Kindel

Updated 1-Oct-19 2:18am

v3

Add a Solution

3 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Richard MacCutchan · Answer 1 · 2019-09-28T04:50:00

Solution 1

Take a look at VMMap - Windows Sysinternals | Microsoft Docs[^], which may be useful.

Posted 28-Sep-19 4:50am

Richard MacCutchan

Gerry Schmitz · Answer 2 · 2019-09-28T07:13:00

Solution 2

If your working set is larger than real memory, then you have paging (to / from virtual [disk] storage).

If your memory access is "all over the place", then you get "more" paging.

If your paging becomes excessive, then you get "thrashing", resulting in performance degradation.

Posted 28-Sep-19 7:13am

Gerry Schmitz

Comments

Alexander Kindel 28-Sep-19 17:32pm

Does memory that is reserved but not committed count as being part of the working set? According to https://docs.microsoft.com/en-us/windows/win32/memory/working-set, "the working set of a process is the set of pages in the virtual address space of the process that are currently resident in physical memory" which, according to https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualalloc, reserved memory is not: MEM_RESERVE "reserves a range of the process's virtual address space without allocating any actual physical storage in memory or in the paging file on disk." I don't expect the amount of memory I actually commit to be anything out of the ordinary.

Alexander Kindel · Answer 3 · 2019-10-01T02:18:00

I tried writing a program that does nothing but reserve some memory then block - I initially tried reserving half the difference between lpMaximumApplicationAddress and lpMinimumApplicationAddress. Then I ran the program and found it in Task Manager. The memory column said 128.3MB. I tried halving the size of the reservation and running it again; Task Manager then reported 64.3MB. Halving the size of the reservation a few more times halved the value Task Manager reported each time. So it would seem that Windows does do something like enumerating every page, even just for reservations, and the people who warned me not to make large reservations were right to be wary.

I made a puzzling observation on top of that: I tried bumping the reservation up to its initial, very large size, but adding a VirtualFree call on it immediately afterward. The value reported by Task Manager stayed steady at 128.3MB rather than dropping back down after the free. So apparently if memory gets tied up in tracking a reservation at any point, there is no way to get it back.