My name is albert burbea, I am a driver programmer at my company and Iahve a nasty question
I would like to limit physical memory access to Windows, using the MAXMEM flag in boot.ini, and I would like then to access this memory mapping it as (for example) a PCI card. Do you have any ideas at how can I do it?
The question is : how do I know which is the maximum physical address windows is able to access?
albert again ....
As you may remember, I am trying to hide the upper physical memory from windows and then allocate it to my driver
I succeeded to obatin the top of memory by GlobalMemoryStatus, but now I do not really know how to go on. The problem is that I am working with windriver tools suite (www.jungo.com) and they sugget to do WD_CardRegister with the phys address of the memory and the size. I may obtain 10 Kbytes of memory this way, but if I try something serious (10MBytes), the code crashes
This is waht nalyze -v returns to me:
An attempt was made to access a pagable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arg1: 00000004, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, value 0 = read operation, 1 = write operation
Arg4: 804e6617, address which referenced memory
***** Kernel symbols are WRONG. Please fix symbols to do analysis.
READ_ADDRESS: unable to get nt!MmPoolCodeEnd
unable to get nt!MmSpecialPoolEnd
unable to get nt!MmPagedPoolEnd
unable to get nt!MmNonPagedPoolEnd
unable to get nt!MmNonPagedPoolStart
unable to get nt!MmSpecialPoolStart
unable to get nt!MmPagedPoolStart
unable to get nt!MiSessionPoolStart
unable to get nt!MiSessionPoolEnd
unable to get nt!MmNonPagedPoolExpansionStart
unable to get nt!MmPoolCodeStart
(.reboot is mine)
What went wrong? and how can i bypass this ?
PS. To make is simpler: I need to allocate several tens of megabytes of memory, and have it accessible from kernel. The memory MUST be physically contiguous. If you have any ideas, I will be more than glad to hear
you can write me at firstname.lastname@example.org or email@example.com
From my kernel this is what the assembly looks like when I unassemble that location:
804e6610 8b0485d0065580 mov eax,[nt!MmPageLocationList (805506d0)+eax*4]
804e6617 8b4804 mov ecx,[eax+0x4]
Anyway, MmAllocateContiguousMemory/MmAllocateContiguousMemorySpecifyCache can allocate phyiscally continous memory. There are other APIs that allocate this memory "safe for DMA", AllocateCommonBuffer, which do the same but ensure that the memory can be accessed by both the processor and the DMA controler at the same time. Is there really a difference? Perhaps there could be, it could even just be ensuring that the memory is allocated in a physical address range accessible by DMA or even not consuming from non-paged pool (as the previous APIs do) but from a special pool for DMA buffers. I don't know I haven't looked at those APIs to see what the difference is, could be nothing.
In anycase, you mentioned before that you are reserving the memory above what the OS thinks is the physical limit for your operations. Did you also need to access that memory from the kernel or just the device? Is this the memory you are having a problem with or is it just that you're allocating 10 megabytes of standard non-paged pool and not continous memory?
you are really helpful, thanks
The problem I am trying to solve is as follow:
I need to allocate a HUGE contiguous buffer (> 50 MB< even 100MB sometimes), and I understood that operating system limits this. So, I decided to "hide" memory from it and try to use for my own purposes. So, how do I do this? Any ideas? I am not a device driver developer, I am using a toolkit (Jungo's windriver) that should help with this. The problem with the toolkit is that I am not neither able to allocate the memory I need nor to allocate the same amount all the time. I mean that sometimes my application tries to allocate say 16 MBytes, and succeeds, but nothing assures me that at the next run I will be able to get it.
THank you again for your time.
BTW, if you prefer to call me, or have me call you, send me an email at firstname.lastname@example.org
1. You can't use the memory above what you told the Operating System there was available.
You wanted to tell the OS that the memory above a certain address didn't exist so your device could use it. This is fine if your device is the only one using it. The OS won't map those physical addresses into the page tables if it doesn't know about it. Your driver can't access the memory if it's not mapped into the page tables, unlike the hardware you can't access physical memory directly.
2. Why do you need very large buffers?
Generally what happens is that the buffer sizes are negociated and they are usually several small buffers, say 5 buffers of 64k for example. These are then used in sequence between the hardware and the device drivers. The device driver can fill up 64k of memory and let the device use it while it fills up the next, or vice versa. Applications running in user mode don't access this memory directly either, they usually issue a request to the kernel to recieve these data buffers and the driver then takes the memory and fills up the user's request.
So you should be able to allocate small physically continous amounts of memory. What is the purpose of the large buffer?
we are using a standard network interface card as a frame grabber. We designed a camera with a GigE interface, and recevie the data using the NIC. The problem is that we send sometimes HUGE images (11 MBytes!) and we need it continuous. MOre, the image is preceeded or followed by data. Also, for other reasons, we need to configure the NIC in a way that it should be able to build correctly the image in memory. Of course , we may need SEVERAL buffers like this, and this is the reeason why I need such a huge buffer. I was wandering if I can map the hidden memory as a sort of a "card" in the PC, and then make reads and writes to it.
Anyway, I really need to know the base address of this hidden memory.
Since physical memory is linear, if Windows thinks there's 32 Meg on the system and there's really 64, then you just need to start at the physical offset of 32 megabytes. The following API can be used by devices to map their memory ranges into virtual memory:
I would attempt to give those a read and attempt to experiment with them. I usually don't work directly with hardware so I can't tell you what the pitfalls are going to be. The other alternative, it's just an idea I had, if those don't work would be to just simply create MDL's and modify them yourself to point to your physical memory and attempt to lock them. It may or may not work but MDL's generally just describe physical pages in memory.
Here are MDL functions which you may need to use them anyway for other purposes: MDL Functions[^]
There is something else you may want to know that different types of devices may have their own "API" for mapping device memory to physical memory. Certain drivers have a framework and most of the APIs available to them (such as video and ndis network drivers) are done through calls to functions that are basically wrappers around the system API. Some are direct call through while others do slightly different operation.
The following examples are how VideoPort drivers map AGP and Video memory to physical:
This is just an FYI if you start looking around for ways to map physical to virtual memory. If you are programming for a particular driver framework you may want to check it out to see if it supports physical to virtual memory mapping. Although, again it may not be best to use it if they do something "special" for that device since you are actually mapping RAM and not device memory.
Here's another thought though, if you have a device that requires 10 Megabytes of memory to perform transfers, why not just put 10 Megabytes of memory on that device and then you would have no problems mapping it or reserving it from RAM?
its Albert again...
About your questions:
1. Yes, I need access from both the kernel, the device and the user
2. I need the memory to be continuous - this is because of the data we are receiving.
3. Where is the memory really beginning? I mean, there must be place for PCI boards and other system peripherals in the PC. Where the phys. memory really begins ?
I wrote the Device Drivers for PCI Card in C Language.I used only I/O Mapping only.The driver is working fine.Now i want the same driver to be work in PCI Express Card.so i need to change the code to Memory Mapped I/O.
Can you suggest me how to solve this problem?I am using WindowsXP/Vista/7 Os
First of all i must say sorry about my bad english, but i'll try to mannage
I'm reading your article 4 the drivers.
I have a task to make a driver for a laser and i have problems.
First problem is that this is new 4 me and i've never wrote a driver before.
How do i prepare a picture 4 the driver to understand?
The laser should be able to burn a picture on wood or metal or stone, but how do i prepare the pictures information 4 the driver? How does the procedure go?
This I think is enough 4 the beginning
I'm new at this and may be commpletly ilogic, but i need all the help i can get
I hope that u can help me with this problem.
Looking forward to your reaplay.
Printer drivers are simmilar to display drivers. Most printer drivers these days run in user mode and call Eng* APIs (Which allows the driver to actually run in user mode or kernel mode since both sets of those APIs are available). You should visit MSDN:
I am finding you driver tutorial to be very well written and helpful. I have a question about your first tutorial. I was able to get the driver compiled and was able to load and test it withe the other programs. In order to compile the driver I had to replace the macro call TYPE_ALIGNMENT(char) with the constant 1. If I left the macro in I was getting the following errors:
error C4116: unnamed type definition in parentheses
I built the driver in the XP Free environment and I used the bld and the default makefile along with a simple Sources file.
So excited to talk to you.
Can I ask you some questions?
Since I want to replace the rdpdd.dll which worked well in windows terminal service, could you tell me how can I do and what will I?
I know the display device of one window is determined by the "default Desktop" of the windowstation. Can I change the Desktop's display device?
Or how Can I create a Desktop with special display device just like rdpdd.dll?
I met another problem:it cant work after installed.
Debugger shows (in appmonitor) hFile==NULL:
hFile = CreateFile("c:\\video.dat", GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, 0, NULL);
A session can have multiple desktops associated with it however only one can be active at any one time. Generally, all desktops will be sharing the same display driver there is no reason to associate different display drivers with different desktops. However, a display driver for a session can be changed provided it is a local session. I.E. you can go into display properties and change the display driver. Remote is a different type of session and is initialized differently or when using FUS is switched into this mode.
In that instance there really is nothing "special" about RDPDD.DLL on the outside as it is just a registered display driver for a particular type of session. You can write any display driver you want and install it to replace the console display driver if you want. The difference in TS is that the graphics are remoted instead of display locally. The driver development article 6 shows how to create your own display driver and use it locally as a multiple monitor setup.
Thank you for your response.
In fact, the work I want to do is to implement a virtual display driver just like RDPDD.DLL except for a different protocol(not TCP/IP, not RDP). So I am confused about how to do.
Is it evitable for me to create a remote session with a deamon service just like MS Terminal Service?
Thank you for your advises.
From my view, VNC is usually used to monitor the console session, so, I think it cannot satisfy what I should do.
Could you please tell me what is a shadower and give me some links to them ?
From my view, VNC is usually used to monitor the console session, so, I think it cannot satisfy what I should do.
That is exactly what a shadower is. It mirrors an existing display to another location, so both locations see the same thing. You could eliminate the console view by disabling the monitor or something so it only goes to one place however you still would be doing essentially the same thing as VNC. Doesn't matter if you do it on the console or the rdp session, it's the same thing you would be required to have an RDP connection or a console logon. You can then scrape the screen or whatever you want to do. However, again if you want to do this independent of those scenarios you would need to follow route #2.
Thank you very much.
I just have tried to replace the console's display with a virtual display driver, and I succeeded.
How can I replace the RDP display device?
It seems there is no way to achive this in "Control Panel".
I'm getting ERROR_NO_MORE_USER_HANDLES eror cocde if my application ruins for long time. so I guess that it should be problem of unclosed handles andI found you article on the same.
I just read your article to find out the "Handle leak", I find it very usefull ,I tried the example you have explained using "Notepad" but I faced some problems while debugging.
I'm using Windbg 6.9.
can you please tell me how to do "Kernal Debugging" , I had select "Kernel Debug" from File menu , click on "Local" tab and "Ok", thn a command prompt comes as "lkd>" .
Does this process is right ?
Then I started "Notepad" and open a file "c:\Example.txt" .
I entered folowing sequence of command and get their output
lkd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
Hi Toby. I just checked your webpage - you have certainly led the developer life I'd wished for! Anyway, I just wanted to ask you a very quick question before I try to absorb your very cool debugging tutorials.
I have an application I have to maintain that is causing Windows to lock completely - ie. the mouse pointer can no longer be moved.. nothing else can be done except powering off the machine. Two questions really:
1) Can mutex/event/semaphore code cause this? (I'm think "yes" because I have the feeling I fixed a problem like this once before related to deadlocking with MFC CSingleLock objects)
2) Will reading your debugging tutorial #7 (and its prerequisits) put me in a position where I can identify this problem?
BTW, I have never really written an x86 assembler code before (done heaps of 680x0 code though).
Yes, locks can cause this problem but not all locks it would have to be a specific lock that would block some aspect of these code paths. That would mean somehow blocking the thread which handles input or blocking the drawing operations of GDI, which both are possible. So if GDI is locked or the display driver is locked then you would never be able to draw or see any user feedback. This is more common then blocking the input thread.
Other possible causes could be using up all the GDI objects, corruption in the display driver. You could even have a thread which sets itself to very high priority and then starves the rest of the system. This commonly happens with audio applications.
The debug tutorial #7 should help you with the locks. I would setup a kernel debugger and first try !running -t to see what all the CPUs are doing. See if they are running in a thread or if they are idle. You can also do !ready to see if any threads are waiting to run, perhaps there are a few threads waiting to run but they are lower priority. !thread -1 ff should show you the current thread of the processor you broke into and the priority will be listed. !locks -v should help you search for locks which may be blocking the system.
Hi. Were you debugging on win9x, i can't see even one debugger wchich will allow to do this effectively, like check on what thread is waiting, any debuger displaying handles, tried to use procexplorer to show handles with waitforsingleobiect,, not mentioning symbols for win98 avability
Windows 9x isn't the best platform to be debugging. The last time I checked a few years back, the WINDBG from MSDN actually worked you just had to copy it over (the install wouldn't work directly on the machine). It wasn't perfect, but you could at least set some break points and do some stepping of code. Some of it was slow though and you couldn't force a break in. The debug APIs are mostly the same so that's how the debugger can still at least do some control, but some extensions most likley won't work if they are looking for specific NT memory locations.
You can also try SoftICE for 9x, used to work pretty good as well as there is a debugger you can download from Microsoft for Windows 9x/ME but it does require a seperate machine as it is a kernel debugger. You may also want to try to find turbo debugger for windows or another free debugger somewhere, maybe ollydebug would work? Try http://programmerstools.org/[^] but I've been luckly and in recent years it's been rare that I have had to debug Windows 9x and the last time I even did was a few years back. All this stuff is on NT. I'm so spoiled now adays I shutter at just debugging a 2000 machine and that's still light years ahead of Windows 9x debugging!
Microsoft Vistual Studio 6.0 also had a debugger you could use in Windows 9x and seemed to work well, but it is kind of a heavy weight application to be installing on a test machien to debug, but it should also work for you.
I would be interested in knowing what Compiler the author recommends be used to develop the examples in these articles.
The articles are in C and yet it appears that the current versions of MS Visual C++ cannot compile C language files anymore.
The new MS Visual C/C++ compilers should still be used to compile the only good high level language, C
The articles did use Visual Studio C/C++ 6.0 compiler however a few of the articles have recently been ported to using Visual Studio 2005/.NET. The only problems that people encountered were new command options that needed to be used and old command line options that needed to be removed. Also, for driver development depending on the environment you setup the compiler may add libraries which would make the executable not loadable in the kernel. Using the DDK environmnet instead should avoid this issue.
I am a firmware guy who mostly uses the Windows desktop as a development cross compilation platform.
Thus, I have yet to figure out how to force the current versions of Microsoft Visual C++ to compile straight C.
Could you describe step by step how your examples could be compiled using the current Microsoft compilers or point to an article here which has already addressed this issue.Thanks.
The environment is simply the command line. Files named ".c" should compile as C. You can run "vcvars32.bat", make sure the makefiles contain the correct path location to the DDK source and then compile via "nmake clean" then "nmake". YOu may also need to "mkdir bin\symbols" in the bin location as it was not apart of the makefiles.
You can also get rid of the "rebase" line in the makefile. Here are some links to other discussions on compiling:
So the C compilation capability is still there in the current MS Visual C++, but not fleshed out in the GUI.
One has to write one's own Nmake to activate the C compiler.
Does this means that for C compilation, the developer simply does not use the current MSVC++ GUI at all, and hand-types everything into his own array of makefiles and batch files ?
The GUI should still work, just create your files as ".C" and probably just have to select "C++ Project". So while the option to create a "C" project is not there, it's likely just implied with "C++" provided you name the files appropriately. THe only problem would be if they default some of the command line options inapproprately for C++, which would then just need to be changed.
I do not use the GUI and have not for a long time though. I like to use makefiles and build environments and work from the command line. It is much easier for me this way to build large projects and independent projects together (I just do nmake from top directory). The build environment I had was very simple I am actually thinking of switching to the latest DDK build environment, would be easier for other people to setup as well then.
Mostly I just use makefile, very little batch files unless really nesecary. So just do "nmake" and build the project. Any batch files would likely just be called from the makefiles anyway and not exposed to the developer. The batch files would also be dependent on the build environment, for example sometimes people use "perl" or "VB Script" to get called from their build environment or other exe files to do special things like build installation and setup programs, call install shield, or whatever they need to do.
Thanks. I printed out and am about to read 5 of your device driver articles which appear very instructive so I just wanted to have that item ironed out beforehand.
This issue of Choice Of Compiler for various development intents and purposes would be a great article for some of the industrious people here to write.
Microsoft, Intel, Borland, Watcom etc.
However, I think you gave a strong recommendation that this work should be approached with not just Microsoft, but the most recent Microsoft.
I actually don't make any reccomendations for the compiler you use and actually when you use the makefile environment the compiler should be quite abstracted from the development. The developer can use whatever editor they want (I use SlickEdit for example) and then the build environment uses the correct compiler for the job. For example, I used an development environment before where some binaries were compiled using Intel (for optimiations of a particular componet) while majority of the project was done using VC. The makefiles just used the correct compiler. The developer also was pretty unware when the environment was ported from 6.0 to 2005 and it was likely done in a few projects at a time.
As for Borland and Watcom I really haven't used them or seen them being used since the DOS days. The makefile environment I have is again quite simple and in the makefiles I actually define directly the compiler and options. In a true environment the compiler is apart of the master build make files and the make files being used by the developer are attempted to be more independnet of being tied or defining the compiler (so the entire project cna be redefined much easier).
So, the articles I have were done in VC 6.0 and this is likely the easiest to use since they were made specifcally with that in mind. However you should be able to change the makefiles to make them build for any compiler with a little bit of work (Since I do not have a sophiscated build environment to make things more abstract).
One more item: what is the best way to obtain or generate a compendium of all the MS Kernel Calls to some of which your articles refer ?
I find that when I try to print these pages Duplex, multi-pages per sheet, that it does not work and wastes paper.
Is there a downloadable list of these MS System Calls in PDF form somewhere ?
hello. this is a student in South Korea.
i am on a project to study.
it is a software security project which encrypts the exe file of a target software, so someone wouldn't be able to crack the software,
and it also solves the license problem.
here it's how it works.
first, it encrypts the exe file, so the PE structure of file is broken.
second, it merges a module which is going to check the user license and decrypt and run the target excutable image.
this looks like this right below.
when it's encrypt and merged, let's imagine the name of file is "calc.exe" which is the same with the name of target exe file.
third, the software is packaged and sold, so an end-user install the package and contacts to the website, gets a certificate.
when the certificate is given, another module encrypts calc.exe with MAC-address, the certificate and the PE image already encrypted before again.
fourth, end-user runs calc.exe and the module is going to be run, because the PE headers of the module is located at the front of calc.exe.
the module copy the target binary on hdd to memory and decrypts it.
then it checks MAC-address, the certificate and create a process from the decrypted excutable image.
the problem is this.
i don't have any knowledge to create a process from an image on memory.
win32 api function "CreateProcess()" doesn't provide that kind of way.
it needs the file path to create a process.
so, if you have any information or knowledge to create a process from an image on memory, or other ways that can solve this problem.
please reply to me.
to e-mail me use this address, "email@example.com".
thanks for reading this all.
-- modified at 20:57 Friday 24th November, 2006
-- modified at 20:59 Friday 24th November, 2006
-- modified at 20:59 Friday 24th November, 2006
-- modified at 21:01 Friday 24th November, 2006
-- modified at 21:02 Friday 24th November, 2006
-- modified at 21:04 Friday 24th November, 2006
-- modified at 21:05 Friday 24th November, 2006
Last Visit: 31-Dec-99 19:00 Last Update: 27-Mar-15 8:30