Disclaimer: The article's content is taken from the openRCE article. (www.openRCE.org). Another pointer is http://www.nynaeve.net/?p=39.
This article will walk you through all the major calling conventions used in Win32 x86 environment. It will also explain the basic differences between all of them and the type of functions where each one of them is best suited.
There are three major calling conventions in modern Win32 x86:
All of the calling conventions have the same set of volatile registers: eax, ecx and edx. All three calling conventions use eax for 32-bit return values and eax:edx for 64-bit return values.
__cdecl is the default calling convention for Microsoft C/C++ compiler. This is an entirely stack based calling convention for parameter passing; the caller cleans the arguments off of the stack when the call returns.
- Variadic Functions: These functions have a variable number of arguments, which is not known at compile time of the callee. So,
__cdecl is useful for these functions because argument-disposal is left to caller who does know the number of arguments at compile time.
- Old-Style C functions without prototypes: Unprototyped functions are treated as variadic functions, because compiler doesn't know whether the function takes a fixed number of arguments or not.
__stdcall has the same semantics as
__cdecl, except that the callee cleans the arguments off of the stack instead of the caller.
- Library Functions: Except the C runtime libraries, virtually all Microsoft Win32 libraries use
__stdcall. The main reason for this is that for
__cdecl functions, the caller typically needs to adjust the stack pointer after every "call" instruction to a
__cdecl function which takes up instruction code space (
add esp, imm8). For
__stdcall functions, we only do this once at the end of the function (
- COM functions: COM Functions uses
__stdcall with the "
this" pointer being the first argument, which is a required part of the COM API contract for publicly accessible functions.
- Functions that need to be called from a language other than C/C++.
- Microsoft-built programs: Also applies to Microsoft kernel mode codes.
- Programs built with DDK: Defaults to
- NT Kernel drivers.
__fastcall passes the first two register-sized arguments in ecx and edx, with the remaining arguments passed on the stack like
__stdcall. If any stack based arguments were present, the callee cleans them off of the stack.
- Leaf Functions: Functions that do not call other functions as the register arguments are passed in volatile registers.
- Functions that do not use their arguments after the first sub function call.
- Short functions that call other functions and then return.
- Functions that interface with assembly code.
__thiscall, implemented by CL, is similar to
__stdcall, except that there is a hidden pointer argument representing a class object ("this" in C++ ) that is passed in ecx.
- Only used if we are using the default calling convention for member functions.