|
Hello all, first time posting here.
At my workplace, we have a requirement to ship a program with as little button pressing as possible. The users on the other end are factory workers and it's well known that if you give them an option, they will misuse it. Anyhow.. the program is going to be written in .NET with WPF. We have to work under the assumption that the machines do not have .NET installed and they can't install it.
Now before I ask the question, which is more of a general "what if", please note, I am not interested in anything but what I am about to ask. For example "oh just install this", or "Oh, not possible". After doing many hours of research, what I ask IS in fact possible, because commercial programs do this (spoon studio, boxed app).
I am an advanced developer, so I know my way into some scary places in the Win32 API and also have been developing in .NET / Win32 / C++ for many years. My current job (past year) has been mainly with embedded C, so I am just getting back to Win32 stuff heavily.
So the subject of the question says it all. I want a standalone .NET exe in which the user does not have to install anything AND of which does not copy any temporary file to the file system. Using the commercial options (spoon studio / boxed app) are not possible, because they are not free, and this has to be a free solution for us.
So here is the meat of what I want to do. I want to make a CLR host that essentially calls into the mscoree.dll to host our .NET application pulling in all the dependencies that mscoree.dll needs. Where do these files come from if we cannot copy any temporary files you ask?: The Dll's are to be stored in the exe itself as resource files.
The Win32 loadlibrary function works only from files on the file system. However, after poking around with Win32 hooking, it has been discovered that LoadLibrary uses ZwOpenFile internally to get a handle to the file you pass in. ZwOpenFile can also not work with anything but a file system file that I am aware of.
Here is my idea. We hook ZwOpenFile using extended code overwriting. After disassembling NTDLL.dll, this is possible, because the prologue to the function is more than 5 bytes. Next, our custom function gets called. Thirdly, ZwOpenFile passes back a PHANDLE pointer to the opened file. How can we rejigger this PHANDLE pointer instead of using ZwOpenFile? How about CreateFile? Using CreateFile, you can pass in a named pipe. CreateFile returns a HANDLE pointer that I ASSume can be assigned to the ZwOpenFile PHANDLE parameter. I don't have a lot of experience with named pipes and what not, but this is just an idea. Maybe there is another way. Basically we get the dll file required by LoadLibrary from the resource stored in the exe originally. Using this block of memory, which is actually the dll itself, we can map it into something so that CreateFile gives us a handle that ZwOpenFile can pass back to LoadLibrary. Now, the big question is... is a file handle created with CreateFile 'compatible' with a PHANDLE that ZwOpenFile passes back. I am guessing that the structure is the same, BECAUSE I also tinkered with hooking the CreateFile API which uses ZwCreateFile internally. Thus, a PHANDLE from ZwCreateFile (which would give us a memory handle to the image instead of a required disk file) should be compatible with ZwOpenFile.
The mscoree.dll also requires some registry keys. I also looked at some registry key functions in a disassembler and it is also possible to hook these. However, I have not worked out the interaction of the registry keys and where they are used yet.
This is just my idea, but hopefully there are some constructive minds out there with more information on the subject. We really want the standalone .NET exe so that we can avoid writing a native Win32 program. PS, I am not interested in using anything other than .NET and WPF: no QT, no mono + GTK, no whatever, so please do not bring those options up.
I would treat this as more of a "what if" question besides trying to worry about what is happening with why I am asking the question.
Thanks
|
|
|
|
|
What you want is possible, but not a single app that can accomplish this is free.
LostTime77 wrote: ZwOpenFile can also not work with anything but a file system file that I am
aware of.
True. You just defeated your own "no temporary files" requirement.
LostTime77 wrote: Here is my idea. We hook ZwOpenFile using extended code overwriting. After
disassembling NTDLL.dll, this is possible, because the prologue to the function
is more than 5 bytes. Next, our custom function gets called. Thirdly, ZwOpenFile
passes back a PHANDLE pointer to the opened file. How can we rejigger this
PHANDLE pointer instead of using ZwOpenFile? How about CreateFile? Using
CreateFile, you can pass in a named pipe. CreateFile returns a HANDLE pointer
that I ASSume can be assigned to the ZwOpenFile PHANDLE parameter. I don't have
a lot of experience with named pipes and what not, but this is just an idea.
Maybe there is another way. Basically we get the dll file required by
LoadLibrary from the resource stored in the exe originally. Using this block of
memory, which is actually the dll itself, we can map it into something so that
CreateFile gives us a handle that ZwOpenFile can pass back to LoadLibrary. Now,
the big question is... is a file handle created with CreateFile 'compatible'
with a PHANDLE that ZwOpenFile passes back. I am guessing that the structure is
the same, BECAUSE I also tinkered with hooking the CreateFile API which uses
ZwCreateFile internally. Thus, a PHANDLE from ZwCreateFile (which would give us
a memory handle to the image instead of a required disk file) should be
compatible with ZwOpenFile.
Blah, blah, blah, ... you just got yourself into more work than it would be to write your own linker (remember the apps that are never free that do this!)
Really, think about this. You said it has to be free, but how much are you (and supposedly your team) is getting paid by the hour?? I'll be willing to bet that the cost of the linker is cheaper than the time it's going to take to build this off-the-wall solution, test the crap out of it before you ship it, and find out that certain parts of the .NET Framework will no longer work because it, like anything dependent upon Reflection for example.
You're not going to whip this up in a few hours. You're GOING to spend a lot of money on this no matter what route you decide to go.
Oh and, on the side, your resulting executable is going to weigh in at over 200MB, plus.
|
|
|
|
|
It is free. My company is spending no money on this solution. I am doing it on my own time as an experiment, because I am interested in proving people wrong, such as yourself. I honestly dislike all the people who try to hammer down others because they are afraid to try something outside the box that is unsupported.
Writing my own linker? Are you kidding me? For your information, I have already done that. Anyhow.. now that that is out of the way, this is nothing like writing one's own linker. This is hooking API calls. I suggest you read what that is before you go spouting nonsense, trying to compare linking and injecting some code into a dll file. I know the difference.
200MB plus? Spoken like one who apparently knows the size of all the dll files needed. Why don't you take a look at an article that the boxedapp people wrote a while back. I believe it is called "Boxedapp .NET redistributable" or something like that. In there, they give all the dll files needed for a standalone console app. I assure you, my friend, it is much less than 200MB. The dlls required in system32 are very lightweight. Kernel32.dll itself is around 1MB. Adding the dlls for WPF amounts to maybe 20MB. I gathered all the dlls that I was referencing from the GAC and checked.
|
|
|
|
|
Just an FYI. I found the article and I pulled in a little more than half the dlls required by that article just to get the clr to run and also all of the dlls for WPF (presentationcore, presentation framework, system.xaml, system.xml, windowsbase). I pulled in the major ones, such as clr.dll, mscoree, mscorlib, etc. which are huge. The rest of the dlls are just small native win32 dlls that need to be pulled in. 31.7MB. That's quite the far cry from 200MB. I just ran winrar on all the ones I pulled in and Ive got a package with a whopping 10 megabytes for that entire slew of dlls that just took up 31.7MB. Assuming you can unzip into RAM, which should be possible somehow, nothing is wrong with storing a zip in the exe.
|
|
|
|
|
CMurphy77 wrote: The rest of the dlls are just small native win32 dlls that need to be pulled in.
31.7MB. That's quite the far cry from 200MB.
I think you're forgetting the class library .DLL's.
|
|
|
|
|
Which .dll are you referring to? I did say that I didn't have everything in that metric; however, I got the major dlls which were very big. If you are referring to system.dll, I already have that. Either way, I expect maybe a 50MB file after all the files are included. Zip them up and you have a very small file as seen by me going from 31MB down to 10.
If you mean to tell me I need every class dll .NET has to offer, then yeah... the file would be very big. BUT, what makes you think I need to include those in the executable? -> because I don't. I only need to include the ones I am using.
|
|
|
|
|
CMurphy77 wrote: you mean to tell me I need every class dll .NET has to offer, then yeah...
Nope. But, you'll find out along the way which ones you're missing.
|
|
|
|
|
I might be able to relax my requirement of no files on the file system. After looking at how windows actually locates DLL files, it first looks in a known DLL directory (system32 for example). Then it looks in the directory that the process was executed from, etc.. What I am thinking is that if I just package all the required DLLs and plop them into the directory as hidden files or something, windows will be able to locate all the DLLs required for the runtime. Next, after peering into how the clr loads, we have some registry keys to sort out. Adding keys to the registry is very easy to do. As far as I can tell, the important key is the install location for .NET for the clr to load the .NET libraries it needs. We can point that to just about anywhere in the key.
So to summarize, We find all the required dll's via trial and error then we zip them up to make the resulting exe small. We package the zip as a resource. The exe itself will be a clr hosting shim. If the system does not have .NET X.X installed, we unzip the zip resource, copy those files to the local folder, and install the registry keys .NET requires. The program then just runs as if the .NET framework were installed. Of course, we can make the program fairly robust making sure all the known files are on disk, locking them while in use, etc. etc.. using the native C++ shim.
This seems like an extremely easy option to package our exe as a "standalone" exe. The resulting exe should not be very big either, considering the metric that I found earlier.
|
|
|
|
|
CMurphy77 wrote: This seems like an extremely easy option to package our exe as a "standalone"
exe
If it was that easy, why isn't everyone doing it?
Look. You're asking a question that nobody is going to be able to walk you through. This is going to take a ton of research on your part. If this is a hobby, great! You've got all the time in the world to work on it and I look forward to seeing an article on the technique.
But, you originated this as a business requirement for deployment, not a hobbyist question. That means you've more than like got a deadline. In all seriousness, I don't think you're going to figure this one out before then.
I'd love to help you, but I simply don't have the time to start a major research project like this.
|
|
|
|
|
There is no deadline on this as of yet. I would suggest you stop making assumptions. The actual starting point of this deployment is way down the road, not anywhere in the near future. You don't have to make it look like you care about any deadline that doesn't actually exist.
On that note, this is not a "hey do this for me". All I was intending was for constructive comments on the situation. I didn't ask for your help, nor do I want anyone to do this for me. As I said, I am experienced developer myself, therefore I can do things myself.
At this point, this is a hobbyist question.
Also at this point, friend, I have implemented the above test by pulling in all the required dll files into a folder and tricking windows into running .NET without it actually being installed. It's very simple. There are two things to note. One is the %windir% environment variable that gets passed into a process. You can start one process and then, without changing the variable, can start another process with a variable that is rejiggered. You can trick the runtime into using your path's instead of its own. Second is the registry key install path for .NET that has to be remapped.
I made a demo WPF application and ran it without .NET being installed. Now, before you try and say it's going to take a long time to do... It took me about 1 hour of probing and setting it up.
Have a nice day. This has been an exercise of proving an experienced developer incorrect.
|
|
|
|
|
I'm glad you had fun! Have a nice life!
|
|
|
|
|
Yeah, thanks for telling me nothing could be done. It really urged me to prove you wrong in a little over an hour. I really appreciate it.
|
|
|
|
|
I didn't say nothing could be done. You just shot me down at every turn.
Have fun being the new "big man on campus".
|
|
|
|
|
I shot YOU down? After I made my first post you spurted out a bunch of "blah blah blah" comments. All you have done this entire time is say "it will take too much time, dont reinvent the wheel, etc." You also incorrectly compared the type of hooking I wanted to do to writing a linker. I don't think so. Get your facts and your head on straight before telling me I shot you down.
This is why I tend to stay away from forums. I am a master of google and if I have to go onto a forum for constructive comments, it because there literally are no resources for what I am doing. I know it is fact that when I ask a question, some person just like yourself is waiting at every turn. You are an element of the system, nothing more or less.
|
|
|
|
|
Hey, if you want to tell me I suck at this, fine. You're throwing your skillset in my face, so use it! Start answering questions in the forums. It's that simple.
By the way, everyone's skill set is different. You're stronger in areas than I am and I'm stronger than you are in others. You'll find the holds true for everyone just by answering questions over time...
If you don't want to answer questions in the forums, don't go waving your skill set in anyone's face.
|
|
|
|
|
I didn't explicitly say you sucked at anything. I did however wave my finger at you, because you made sweeping accusations before knowing what was going on. For instance "your exe is going to be 200MB+". Or "you are GOING spend xx amount of time-> how much do you get paid an hour?" I also made a note at the end of my question that read "Please do not read into 'why' I was asking the question. This is because I didn't want to tie it to my work situation, even though that is my end goal. The intention was to experiment on my own time for fun. I dislike it when people come in thinking they know things, when I myself have done the research before even posting in the thread. I spent literally this entire week researching this topic, because it is very important to me. I knew the size numbers even before you tried to post what you thought.
I apologize for making it seem like I was waving things in your face, but your first post was taken the exact same way. I thought your first response was aggressive and I even poked into your profile to see what your experience was before responding. I was surprised to see you are an experienced developer and you threw out the accusations like nobody's business.
The reason you don't see me on the forums is the same reason I rarely RARELY post on forums. As I said, I only ask questions when I am at a dead end and I have tried so many combinations on google wherein the first 10 pages with links all show up in purple. That is how much research I have done.
|
|
|
|
|
And I apologize for lumping you in with the stereotype OP based on the question you asked.
|
|
|
|
|
In case anyone is interested on the status of my topic. After several more days of probing, I have successfully been able to run a WPF application by unpacking a much smaller .NET distributable compared to having to install the entire framework. My app is simplistic, and just is a WPF window with a button inside that when clicked shows a messagebox. I wanted to just get this working and test the theory. I have packed all the necessary files: registry entries, .NET framework folder, and my simple WPF app into a file. After purging the .NET framework folder of everything unneeded (dll, install files, configuration files), the resulting distributable folder is 48.1MB. Using 7z to pack this results in a file that is 12MB. This includes system.dll, PresentationFramework.dll, PresentationCore.dll, System.xaml, and system.xml managed assemblies. Also, using the method I have devised, you can unpack this distributable file to any location and run the application from there. This will come in handy when I elaborate on the end goal in the next sections.
So let me take a step back and explain why I did this. Here is the scenario. We want to deploy our application to manufacturing; however, the computers that are being used are running a minimum of XP. This means that we have to plan for no .NET framework install on the system. Keep in mind that the people running our application don't understand anything about installs, etc., therefore we have to make it a click once solution. The most that they know how to do is download a file from the internet, copy it to a flash drive and run it on the computer.
With this in mind, we of course can say to them: hey, install the .NET framework first then use our program. This introduces another step for them. Believe me when I say introducing steps into this system is a bad idea. These people are computer illiterate and do not have the time to be tracking down files and installs. The less stuff you have in the system the better. We also have the goal of having a custom branded UI the entire way. Lastly, we want to run a few custom program / build steps before we actually install our program to verify certain things on the computer beforehand. In the end, we will install the .NET framework 4 if it is on the computer, BUT, we also want this entire process to be inline with our custom branding. Thus we cannot run the .NET installer as is. Luckily, the .NET installer allows us to install silently and also pipe progress to our program via a memory map. Also, keep in mind that the .NET installer will take several tens of seconds to fully install on the system.
So.. here is the series of events that will happen: User downloads our click once file and runs it on their computer. We get and create a temporary directory. If .NET is not installed on the system, we unpack our .NET redistributable. We then unpack our custom WPF app. Additionally, a standalone version of 7zip is unpacked to the directory so that we can unpack stuff. In order to run the WPF app without .NET, we need to install some registry keys. They get installed. Next, our WPF app is started. It then deletes the registry keys that it installed. Since the WPF app is already started, it can do this, because the registry is not needed anymore. The user then presses next to install our real application. During this process, since .NET is not installed, we unpack the .NET installer and pipe the progress to our WPF app and install the framework.
With this method, the .NET installer has no knowledge that we are in fact running a WPF app without the framework installed, because the registry keys used are already cleaned up, and we are running the framework from a temp directory. Of course, the above method is watered down, but if anybody is interested in specifics, I can explain further.
Now, the reason it took so long to figure this out was because I was missing a piece of the puzzle. It turns out that the .NET framework uses TWO important registry nodes to get directory information. This is how you can run the framework from a directory outside of C:\Windows\.NET.... etc.. The two important nodes are HKLM.....software\microsoft\.NETFramework AND HKLM...software\.NETFrameworkSetup... and all associated keys. There are I think 4 path keys that point to directories for the framework. If you redirect them all, to your custom framework folder, you can run the framework from the custom directory.
Next, there are a few more things we have to realize. The framework needs a set of native dlls to run that are installed with the framework into the system32 folder. I believe we can also reroute these dlls into our custom .NET folder and have it work. As I write this, I kick myself for forgetting to test that aspect out. Anyhow, the important dlls are mscoree.dll, mscoreei.dll, mscoreeis.dll, msvcxxx_xxxclr.dll. I don't know the exact list off the top of my head, but as long as you have these in a path that can be found by distributable, it will run fine. These dlls are unpacked with the .NET framework and are not shipped with a bare install of XP. Windows 7 might already have some of these dlls due to .NET already being installed. In addition to the above dlls, you need to also put mscorwks.dll into the .NET distributable framework folder.
Lastly, it turns out that if the clr cannot find the managed assembly that it needs, it actually falls back to looking in the directory where you started the original exe from. The clr uses the registry to determine locations of assemblies, such as the GAC assemblies. I couldn't be bothered to track down these keys or figure out how they worked and solely relied on the fact that the clr will look at the directory you started the exe in. Therefore any assembly you need for the program can safely be packed in the same directory as the program itself and be found. When I was testing, the clr tried to look for GAC assemblies using the registry, and of course they were not in the registry, because I tested my method by uninstalling the framework first.
So overall, this entire process should take around 3 - 5 seconds to execute on a slow XP machine without the .NET framework. So within that time you can copy the files, modify the registry and execute a WPF program without installing the framework. If anybody is actually interested in more detail, drop a line.
|
|
|
|
|
Drop a line? Heck, I'd love to see an article on this. It could, potentially, be the most interesting article of the year.
|
|
|
|
|
Haha, glad to see a person with jesus status is interested. Unfortunately, I have not worked out all the kinks yet such as the placement of the system32 dlls. I believe I tried them in the framework folder at one point and it worked, but I can't remember. Today I started from scratch and tested the WPF portion, because ultimately, we want a nice WPF UI to be able to run without .NET installed. Also, there is the issue of the wow6432 node as well.
Overall, I have to make essentially a shim program that does all of the above and checks various things on the system, one of them being if the system is 64 bit and putting the registry entires in the proper place. This will be done in native win32 c++. However, since all we are doing is modifying the registry, unpacking and copying files, and run processes through CreateProcess, this shim should not be hard to do.
Once I get that up and running with all the needed components, I could write an article. Keep in mind that that all sprouted from the fact that we wanted a custom branded WPF UI on a computer without the .NET framework installed. The user can of course spend 20 seconds installing the framework, but then they see the crusty default win32 installer UI. We wanted to make it so that the entire process of installing our program AND the full .NET framework resided in our custom branded UI. If the user is willing have a delay of 5 seconds before they see our screen due to the fact the framework is not installed initially, I think that is acceptable. 5 seconds is relative to tests that I did awhile ago with extracting 7z files into a folder. It could very well be much shorter on a modern SSD based computer. I am testing on a very old 5400RPM harddrive core 2 duo system with XP on it, because that is our base metric.
It turns out that its very simple to do all the things that I once thought were magic such as getting your program to show up as a program in windows and the control panel. Its 2 registry keys. Add a few more registry keys and now you are on the right click context menu on the desktop. I think I have learned more about how the windows registry works in these past couple of weeks than I have since I became a programmer more than half a decade ago. It's also more than I wanted to know too.
I was throwing around various ideas in order to attempt to meet our requirements. My first idea was to write the program in mono and a UI in GTK to do what we wanted. This is because you can distribute a standalone exe and have it work. However, mkbundle sucks on windows, and I posted several problems in trying to get that work on their forums. Of course, nobody cares on their forums. On the bright side, you can package a few dlls with your exe in mono and it just simply works. This is because mono does not rely on the windows registry at all and will look for the dll's it needs in a defined directory. You need to define a lib folder and a root folder and that's it. Even though to get the mono solution working was easy, I just kept coming back to: "Nothing can beat WPF, nothing can beat WPF". Its true in my opinion. After trying out GTK for a few days, you realize how awesome WPF really is with its layout system and just everything about it.
The next idea was to use silverlight. The SL installer takes about 9 seconds on the machines I tested with. With this option, I was planning on making a bare simple mono GTK UI with the method described above that just popped up a custom branded windows that told the user "Please wait, installer initializing" while silverlight silently installed in the background. Then I could use silverlight with the power of WPF to make our custom branded UI after that. This turned out to be doable, and I even tried to reroute registry keys so that I could run silverlight from a directory other than the install directory just like I accomplished with the .NET option. Even though I got my application to run, there are problems with this mechanism. The first is that its hard to pass command line arguments to a silverlight program. The second is that now we have 3 technologies working in our system: mono, silverlight, and .NET when it comes down to installing the real program. A little bit of a head scratcher when trying to figure out what is going on. I would have to tell my boss that it would be hard to wrap his head around what was going on. Therefore, this option is out.
Finally, with my method described with the .NET redist, you only have a single framework: .NET and WPF AND you can pass command line arguments to the program. The program has full trust and a powerful UI framework to go with it. Any control libraries I make for the .NET redist in WPF I can then reuse in my real .NET app that we are packaging with the installer.
|
|
|
|
|
CMurphy77 wrote: My company is spending no money on this solution
So you work for free?
|
|
|
|
|
Question for you. Do you like attempting to short circuit conversations for no purpose other than to make cheap shots and make people look bad? Why don't you focus on the situation at hand, and if you can't do that, then don't respond in such an aggressive, pointless manner. I am not interested in your 'many years of experience' in saying things are not possible just because they are inherently unsupported, meaning nobody has tried them: not that they cannot be done.
Secondly, to answer your question directly... No I don't work for free. I am looking at the packaging option presented here on my own time. One does not need to be paid in order to dabble in a hobby.
|
|
|
|
|
CMurphy77 wrote: in saying things are not possible just because they are inherently unsupported
Oh, no, no, no. There are things that are not worth the cost if they've already been done cheaper. There have been way too many questions around here where the person thinks they cannot afford a paid solution, but completely forget to factor in the cost of their own time when trying to "roll their own" because they think it's cheaper.
|
|
|
|
|
Believe me, I work in an environment where you have to have the mindset to not roll your own. The people I work for want results 100% of the time. At any given time I am literally working on 5 different projects. Not because management is scatterbrained, but because they want results. This is completely different. I find this topic fascinating: Trying to make windows do something it was not intended to do.
If the solution I am looking into right now proved to be as complicated as writing a linker or rolling my own whatever, then it probably wouldn't be worth it as you said. I agree with you that many people think they can storm in and write something and realize after the fact it is going to take a year to do. Been there done that.
If what I want to do is as easy as injecting some hooks into some process code to modify the basic operation of a function, then it is worth it. It seems possible.
After taking a look at ZwOpenFile, it looks like it may be able to handle named pipes natively, meaning that the HANDLE the function gives to LoadLibrary may just work. One could load the dll from the exe resource into memory, then shove that in some sort of named pipe. ZwCreateFile would then just create a file HANDLE out of that.
|
|
|
|
|
LostTime77 wrote: I would treat this as more of a "what if" question besides trying to worry about
what is happening with why I am asking the question.
If it were possible, people would abandon the runtime and go for the native option. Anything else? Seems you've already forbidden every remark I can make 
FWIW, there are more runtimes available, and some of them are open-source.
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
