Calling to an EXE function from inside a DLL

You cannot link an exe file in the same way you would a dll. If you need to access functions within an exe I think you will need to have them exposed via COM.

coffeenet says:

How about variables? Is there a way to reach a variable on the EXE side from the dll?

Variables cannot be accessed between a DLL and an EXE as far as I know, you could try

//some_exe_file.cpp
int nMyVar = 0;

//some_dll_file.cpp
extern int nMyVar;

That should work. Otherwise you will need to define a function to get/set the variables
In C++ you could use just 1 function that returns by reference

//some_exe_file.cpp
int nMyVar = 0;
int &GetMyVar() {
	return nMyVar;
}

//some_dll_file.cpp
int &nMyVar = GetMyVar();

Or in C you would need separate get and set functions

//some_exe_file.cpp
int nMyVar = 0;
int GetMyVar() {
	return nMyVar;
}
void SetMyVar(int nNewVal) {
	nMyVar = nNewVal;
}

//some_dll_file.cpp
int nMyVar = GetMyVar();
SetMyVar(2);

coffeenet says:

Btw, do I need to declare those exported functions twice(once with dellexport, once without)?

No. As with DLLs you can use the exported functions inside the module that is exporting them.
There is nothing special to do. The compiler will make any changes needed, and you use them as you would use any other function.

coffeenet says:

I am in a dilemma right now, a catch 22, where functions need exported functions from the exe, and functions in the exe need exported functions in the dll.

You have 3 ways to fix this.
You will need to assess which ways are possible, and which will provide the best results in the long term.

1. Move the exported functions that require the DLL out of the exe into another DLL.
This may not solve your issue, depending on what requires what.

2. Move the exported functions from the exe into the DLL that requires them.
This may also not be viable if these functions must be in the exe for whatever reason.

3. Use runtime linking to the DLL.
You don't link to the .lib file produced when compiling the DLL, and you don't directly call any functions in the DLL.
This is done with LoadLibrary (or GetModuleHandle if the library is already loaded which it wont be in this case) and GetProcAddress.

I have written a small class which simplifies this somewhat if you are using C++ you are welcome to use it.
This is a simplified copy of my code. Mine also included error reporting to logs.

template <typename T>
class DynamicFunction {
	public:
		DynamicFunction(LPCWSTR szModule, LPCSTR szProc, bool bFreeWhenDone = true)
			: m_pFunction(NULL), m_bLibLoaded(false), m_bWantUnload(bFreeWhenDone) {
			m_hLibrary = GetModuleHandleW(szModule);
			if (m_hLibrary == NULL) {;
				m_hLibrary = LoadLibraryW(szModule);
				if (m_hLibrary != NULL) {
					m_bLibLoaded = true;
				}
			}
			if (m_hLibrary != NULL) {
				m_pFunction = (T)GetProcAddress(m_hLibrary, szProc);
			}
		}
		DynamicFunction(LPCSTR szModule, LPCSTR szProc, bool bFreeWhenDone = true)
			: m_pFunction(NULL), m_bLibLoaded(false), m_bWantUnload(bFreeWhenDone) {
			m_hLibrary = GetModuleHandleA(szModule);
			if (m_hLibrary == NULL) {;
				m_hLibrary = LoadLibraryA(szModule);
				if (m_hLibrary != NULL) {
					m_bLibLoaded = true;
				}
			}
			if (m_hLibrary != NULL) {
				m_pFunction = (T)GetProcAddress(m_hLibrary, szProc);
			}
		}
		~DynamicFunction() {
			if (m_bLibLoaded) {
				FreeLibrary(m_hLibrary);
			}
		}
		void Unload(bool bForce = false) {
			if (m_bLibLoaded && (m_bWantUnload || bForce)) {
				m_bLibLoaded = false;
				FreeLibrary(m_hLibrary);
				m_hLibrary = NULL;
				m_pFunction = NULL;
			}
		}
		T operator *() {
			return m_pFunction;
		}

	protected:
		HMODULE m_hLibrary;
		T m_pFunction;
		bool m_bLibLoaded;
		bool m_bWantUnload;
};

To use this class:
This sample will dynamically load the function IsWow64Process[^].
This function is only available on Windows XP SP2 and later.

DynamicFunction<BOOL (WINAPI *)(HANDLE, PBOOL)> dfIsWow64Process("Kernel32.dll", "IsWow64Process");
if (*dfIsWow64Process != NULL) {
	BOOL bIsWow64 = FALSE;
	if ((*dfIsWow64Process)(GetCurrentProcess(), &bIsWow64) && bIsWow64) {
		printf("This is a 32 bit process running on a 64 bit computer.\n");
	} else {
		printf("This is either a 32 bit process running on a 32 bit compuer or\n");
		printf("it is a 64 bit process running on a 64 bit compuer.\n");
	}
} else {
	printf("This OS does not have the function IsWow64Process.\n");
}

The general layout of the function is:

DynamicFunction<return_type (call_convention *)(parameter_type, parameter_type, ...)> dfIsWow64Process("dll_name.dll", "function_name");

Most functions you write will just use the default call_convention, so this would not be necessary for those functions.


If you are not using C++, then you have a bit of work to do.
This example is equivelent to the previous sample but will work in C.

typedef BOOL (WINAPI *IsWow64ProcessFunc)(HANDLE, PBOOL);
IsWow64ProcessFunc pIsWow64Process = NULL;
BOOL bIsWow64 = FALSE;
HMODULE hKernel32 = LoadLibrary("Kernel32.dll");
if (hKernel32 == NULL) {
	//error: dll not found
} else {
	pIsWow64Process = (IsWow64ProcessFunc)GetProcAddress(hKernel32, "IsWow64Process");
	if (pIsWow64Process == NULL) {
		//error: function not found
		printf("This OS does not have the function IsWow64Process.\n");
	}
	//do the same for all other functions that you need to load from the DLL
}
if (pIsWow64Process != NULL) {
	if (pIsWow64Process(GetCurrentProcess(), &bIsWow64) && bIsWow64) {
		printf("This is a 32 bit process running on a 64 bit computer.\n");
	} else {
		printf("This is either a 32 bit process running on a 32 bit compuer or\n");
		printf("it is a 64 bit process running on a 64 bit compuer.\n");
	}
}

coffeenet says:

1- How can the DLL trigger callback functions that will eventually call functions inside the EXE.
The piece of code you kindly provided above:
2- Where should it go, DLL or EXE?

It seems no matter what way you go you end up using pointers to functions.
This looks daunting, but is rather simple.

I will answer 2 first because it is quick.
The code to dynamically load a function would go in the exe. That would then get a handle to your DLL and load functions from the DLL.

Now for 1.
We use typedef to declare a variable type. This is not required, but simplifies things later.
The syntax of a function type definition is

RETURN_TYPE (CALL_CONVENTION *FunctionTypeName)(PARAMETERS)

CALL_CONVENTION is often left out for functions you write as you just use the default.
RETURN_TYPE includes pointers and/or references (references are only available in C++)
PARAMETERS can be copied directly from the function definition in the .h file or implementation in the .c file.
Parameter names and in/out can be removed to compress the size of the line if you wish.

Lets pass the function strstr into another function as a parameter

//The definition in string.h is:
_Check_return_ _CRTIMP _CONST_RETURN char *  __cdecl strstr(_In_z_ const char * _Str, _In_z_ const char * _SubStr);
//We have
//RETURN_TYPE:  _Check_return_ _CRTIMP _CONST_RETURN
//  "_Check_return_" MUST be removed. It tells you and the compiler that the return value is important. This will cause a compile error if it is not removed. If you are unsure, you can try with this first, if it gives an error then delete it.
//  "_CRTIMP" can be removed. This is used to statically link when compiling a DLL.
//  "_CONST_RETURN" resolves to "const". It must be left in or changed to "const".
//CALL_CONVENTION:  __cdecl
//  This is the default calling convention.
//  It can be removed if you want, and if you haven't changed the default convention.
//PARAMETERS:  _In_z_ const char * _Str, _In_z_ const char * _SubStr
//  "_In_z_" can be removed. It is a guide to tell you that the parameter is an input only.
//  "_Str" and "_SubStr" can be removed. The compiler doesn't care what the variables are called.

//If we copy over everything we can, then we get this (notice the similarity?):
typedef _CRTIMP _CONST_RETURN char *(__cdecl *func_strstr)(_In_z_ const char * _Str, _In_z_ const char * _SubStr);
//As far as the compiler is concerned this is exactly the same as
typedef const char *(__cdecl *func_strstr)(const char *, const char *);
//Or, if you are using the default calling convention, it could be written as
typedef const char *(*func_strstr)(const char *, const char *);

Now that you know how to define a function as a parameter type, we need to use it.
The variable type that is defined as a function (func_strstr in our case) can be used in the exact same way as any other pointer, with 1 addition. It can now be used as a function name to be called.
Keeping with the strstr, you would define a function in the DLL like

__declspec(dllexport) const char *FindSubString(func_strstr pStrStr, const char *szStr, const char *szSubStr) {
	//We just use the function variable as we would use any other function.
	return pStrStr(szStr, szSubStr);
}

And to use this function from the exe you would have

//this definition would be in a .h file somewhere
__declspec(dllimport) const char *FindSubString(func_strstr pStrStr, const char *szStr, const char *szSubStr);

char *szStr = "hello world";
char *szSubStr = "lo";
char *szLocation = FindSubString(&strstr, szStr, szSubStr);

In the above we have FindSubString(&strstr, szStr, szSubStr); which is where the interesting bit happens.
The &strstr is passing a pointer to the function strstr into the function FindSubString.
The ampersand symbol & is optional. I prefer to use it as you are passing in a pointer to the function.

If you have understood everything so far you may have realised that this example is completely pointless as it performs the exact same operation as

char *szStr = "hello world";
char *szSubStr = "lo";
char *szLocation = strstr(szStr, szSubStr);

however the string comparison is done in the context of the DLL.

As for "triggering" a callback, it is up to you when it is triggered. Just pass the function pointer for the function from the exe into the function that is in the DLL and call it as I have shown.

The method of passing function pointers around can also be used for passing a function back from the DLL to the exe as well. The exact same principal applies, however you will probably be passing the function pointer back as the return value of a function.