I am writing plugin for Photoshop.
Photoshop exposes C plug-in interface.
I use WPF for graphic user interface, and C++/CLI for an intermediate layer.
In one moment I call DoUI() function from C code. DoUI() is in the .cpp file compiled with /clr switch on. It means that DoUI() is compiled not to native code, but rather to MSIL. MSIL is JIT compiled in run-time when I run my plugin. However, I have a problem, and I believe it is JIT.
Plug-in works perfectly when Photoshop is not allocating large space on RAM. However, when I load large files in Photoshop and apply different memory demanding filters on them, and then try to load my plug-in FOR THE FIRST TIME - Photoshop crashes. If I load plug-in first, and then load large files and use filters - plug-in loads normally. Also, I experimented with different smaller /clr functions - instead of DoUI(). Same happens. Photoshop crashes just before calling /clr function.
So, regarding the facts that:
(1) Photoshop crashes immediately before calling first /clr function when allocated large space on RAM.
(2) Photoshop crashes only when user attempts to load plug-in FOR THE VERY FIRST TIME when large space on RAM is allocated. If plug-in is loaded before large allocations, plug-in usually loads normally after allocations. However, there is no guarantee.
I presume that there is no enough space in Photoshop memory for JIT to compile MSIL code. Photoshop crashes EXACTLY when JIT is called (calling /clr function for the first time). This explains why plug-in can be called after large allocation if it has been called before large allocation. When there are no large allocations, JIT can run successfully, store compiled code, and so the program can run without JIT after large allocations happened.
Does anybody have any advice how to control JIT behavior and what to do in this situation? How to ask Photoshop to prepare space for JIT?
ps. I tried ngen.exe. However I am not sure if Photoshop loads native cache or runs JIT.