Calling Conventions in C/C++
In C/C++ programming, a calling convention is a set of rules that specify how a function will be called. You might have seen keywords like __cdecl or __stdcall when you get linking errors. For example:
error LNK2019: unresolved external symbol "void __cdecl A(void)" (?A@@YAXXZ) referenced in function _main
Here, we see __cdecl being used. It is one of the calling conventions in C/C++. You can also see code like this in 3rd party libraries.
For example:
extern __m128i __cdecl _mm256_mask_cvtepi32_epi16(__m128i, __mmask8, __m256i);
What are caller and callee?
When we call a function, a stack frame is allocated for that function, arguments are passed to the function, and after the function does its work, the allocated stack frame is deallocated and control is passed to the calling function.
The function that calls the subroutine is called the caller. The function that gets called(i.e. subroutine) by the caller is called the callee.
C++
// C++ Program to illustrate the caller and callee #include <iostream> // callee void func() { std::cout << "Beginner" ; } // caller int main() { // function call func(); return 0; } |
Beginner
In the above code, main() is the caller and func() is the callee.
Calling Conventions
The Calling Conventions in C/C++ are the guidelines that determine:
- How the arguments are passed onto the stack.
- Who will clear the stack, caller or callee?
- What registers will be used and how.
Syntax
The following syntax shows how to use the calling convention:
return_type calling_convention function_name {
// statements
}
The C++ code gets converted to object code at the end of the compilation stage. Then we get the object file. The object files are linked together to create a binary file(exe, lib, dll).
Before the creation of the object file, we can tell the compiler to stop and give us the .asm file. This is the assembly file which will get converted to an object file. Different calling conventions produce different assembly codes. For GCC, the -S can be used for this purpose. Just pass this flag while compiling the code.
gcc -S sourceFileName.c
Different Calling Conventions
There are many calling conventions for different platforms. We are going to look at 32-bit x86 calling conventions.
- __cdecl
- __stdcall
- __fastcall
- __thiscall (C++ only)
Example of Calling Convention
The following program illustrates how to use the calling convention.
C++
// C++ Program to demonstrate the calling convention #include <iostream> // __cdecl calling convention int __cdecl cdeclAdd( int a, int b) { int c = a + b; return c; } // __stdcall calling convention int __stdcall stdcallAdd( int a, int b) { int c = a + b; return c; } // __fastcall calling convention int __fastcall fastcallAdd( int a, int b, int c, int d) { int e = a + b + c + d; return e; } // __thiscall calling convention class Temp { public : int __thiscall thiscallAdd( int a, int b) { int c = a + b; return c; } }; // driver code int main() { int result; Temp obj; // Function calls and output result = cdeclAdd(1, 2); std::cout << "Result: " << result << std::endl; result = stdcallAdd(3, 4); std::cout << "Result: " << result << std::endl; result = fastcallAdd(7, 8, 9, 10); std::cout << "Result: " << result << std::endl; result = obj.thiscallAdd(5, 6); std::cout << "Result: " << result << std::endl; } |
Output
Result: 3
Result: 7
Result: 34
Result: 11
Here we have 4 functions: cdeclAdd(), stdcallAdd(), fastcallAdd(), and thiscallAdd() with calling conventions __cdecl, __stdcall, __fastcall, and __thiscall respectively. They are being called by the caller i.e. main(). Let’s understand each of them one by one.
__cdecl
The __cdecl calling convention is the default calling convention in C/C++. In this calling convention:
- The arguments are pushed from Right to Left (so that the first argument is nearest to the top-of-stack).
- Caller cleans the stack.
- Creates larger executables than __stdcall, because it requires each function call to include stack cleanup code.
As the caller cleans the stack in this convention, we can provide variable arguments to the functions with the __cdecl calling convention.
Compiling the above program in Debug mode using assembly code, we get:
main()::result = cdeclAdd(1, 2);
Ruby
line 42 : push 2 push 1 call ?cdeclAdd @@YAHHH @Z ; cdeclAdd add esp, 8 mov DWORD PTR _result$[ebp], eax |
Note: If you use Release Mode, you might not see the variables passing onto the stack like this as the compiler will produce optimized code.
The function with calling convention __cdecl is cdeclAdd() which is called on line 42 in the main function.
cdeclAdd()
Ruby
?cdeclAdd @@YAHHH @Z PROC ; cdeclAdd, COMDAT ; File c:\users\ruchit\documents\code\visual studio\visual studio\source.cpp ; Line 6 push ebp mov ebp, esp sub esp, 204 ; 000000ccH push ebx push esi push edi lea edi, DWORD PTR [ebp- 204 ] mov ecx, 51 ; 00000033H mov eax, - 858993460 ; ccccccccH rep stosd mov ecx, OFFSET __F81044A6_source @cpp call @__CheckForDebuggerJustMyCode @4 ; Line 7 mov eax, DWORD PTR _a$[ebp] add eax, DWORD PTR _b$[ebp] mov DWORD PTR _c$[ebp], eax ; Line 8 mov eax, DWORD PTR _c$[ebp] ; Line 9 pop edi pop esi pop ebx add esp, 204 ; 000000ccH cmp ebp, esp call __RTC_CheckEsp mov esp, ebp pop ebp ret 0 ?cdeclAdd @@YAHHH @Z ENDP ; cdeclAdd |
Explanation: When we look into the assembly code of cdeclAdd(), the last statement is: ret 0. This means that the callee will do nothing to the stack pointer and the control returns to the main().
If we look at the assembly code in main(), after the function call, there is another statement: add esp, 8. When the control is returned to the main, it will add in stack pointer(esp) 8 bytes. Adding in memory means that we are popping from the stack or clearing up the stack. If we subtract from esp, this means we are pushing to the stack. The stack grows in reverse order with respect to memory.
Note: Here we are incrementing 8 bytes in esp because we are popping 2 integer variables passed. Since each integer variable in the 32-bit arch is 4 bytes, we are clearing 8-byte memory for both variables.
This means that the caller (i.e. main) will clear the stack and not the callee (i.e. cdeclAdd).
__stdcall
This is a Microsoft-specific calling convention used by Win32 API functions. In this convention:
- Arguments are pushed from Right to Left.
- Callee cleans the stack.
main()::result = stdcallAdd(3, 4);
Ruby
line 45 : push 4 push 3 call ?stdcallAdd @@YGHHH @Z ; stdcallAdd mov DWORD PTR _result$[ebp], eax |
Like cdeclAdd(), we see that arguments are passed from Right to Left.
In line 42 in the assembly Code, we see that arguments are passed from Right to Left for function call: stdcallAdd(3, 4)
The function with calling convention __stdcall is stdcallAdd() which is called on line 42 in the main function. If we look at the assembly code of main(), after the function call, there is no statement for the stack pointer. It ends with the function call.
stdcallAdd()
Ruby
?stdcallAdd @@YGHHH @Z PROC ; stdcallAdd, COMDAT ; File c:\users\ruchit\documents\code\visual studio\visual studio\source.cpp ; Line 13 push ebp mov ebp, esp sub esp, 204 ; 000000ccH push ebx push esi push edi lea edi, DWORD PTR [ebp- 204 ] mov ecx, 51 ; 00000033H mov eax, - 858993460 ; ccccccccH rep stosd mov ecx, OFFSET __F81044A6_source @cpp call @__CheckForDebuggerJustMyCode @4 ; Line 14 mov eax, DWORD PTR _a$[ebp] add eax, DWORD PTR _b$[ebp] mov DWORD PTR _c$[ebp], eax ; Line 15 mov eax, DWORD PTR _c$[ebp] ; Line 16 pop edi pop esi pop ebx add esp, 204 ; 000000ccH cmp ebp, esp call __RTC_CheckEsp mov esp, ebp pop ebp ret 8 ?stdcallAdd @@YGHHH @Z ENDP ; stdcallAdd |
Explanation: When we look into the assembly code of stdcallAdd(), the last statement is: ret 8. This means that it increments the stack pointer by 8 bytes and pops the 2 variables that were passed to it. Then, the control returns to the main().
So, this means that callee (i.e. stdcallAdd) will clear the stack and not the caller (i.e. main).
__fastcall
In __fastcall calling convention, the arguments are passed to the register if possible.
- The first two arguments are passed in register ECX and EDX. The remaining arguments are passed on the stack from Right to Left.
- Callee cleans the stack.
Going back to the assembly code of the main function, the statement
main()::return = fastcallAdd(7, 8, 9, 10);
Ruby
; Line 48 push 10 ; 0000000aH push 9 mov edx, 8 mov ecx, 7 call ?fastcallAdd @@YIHHHHH @Z ; fastcallAdd mov DWORD PTR _result$[ebp], eax |
In fastcallAdd, the arguments are pushed onto the stack in Right to Left order, then the 2 arguments that remain are pushed onto registers EDX and ECX respectively. (Or we can say the first 2 arguments are passed onto ECX and EDX in Left to Right order)
The function with the calling convention __fastcall is fastcallAdd(), which is called on line 48 in the main function. We have seen how the arguments are passed onto the stack. We know that since the caller i.e. main is not doing anything about the stack pointer, it is the callee’s job to clean up the stack.
fastcallAdd()
Ruby
?fastcallAdd @@YIHHHHH @Z PROC ; fastcallAdd, COMDAT ; _a$ = ecx ; _b$ = edx ; File c:\users\ruchit\documents\code\visual studio\visual studio\source.cpp ; Line 20 push ebp mov ebp, esp sub esp, 228 ; 000000e4H push ebx push esi push edi push ecx lea edi, DWORD PTR [ebp- 228 ] mov ecx, 57 ; 00000039H mov eax, - 858993460 ; ccccccccH rep stosd pop ecx mov DWORD PTR _b$[ebp], edx mov DWORD PTR _a$[ebp], ecx mov ecx, OFFSET __F81044A6_source @cpp call @__CheckForDebuggerJustMyCode @4 ; Line 21 mov eax, DWORD PTR _a$[ebp] add eax, DWORD PTR _b$[ebp] add eax, DWORD PTR _c$[ebp] add eax, DWORD PTR _d$[ebp] mov DWORD PTR _e$[ebp], eax ; Line 22 mov eax, DWORD PTR _e$[ebp] ; Line 23 pop edi pop esi pop ebx add esp, 228 ; 000000e4H cmp ebp, esp call __RTC_CheckEsp mov esp, ebp pop ebp ret 8 ?fastcallAdd @@YIHHHHH @Z ENDP ; fastcallAdd |
Explanation
Here, we can see that, the stack pointer is incremented with 8 bytes only. We are only popping two 4-byte integers from the stack. This is because the other 2 arguments are passed into ECX and EDX registers. So, we only need to pop the 2 arguments that are on the stack.
__thiscall
The __thiscall calling convention is the default calling convention used by methods inside a class. That is why it is only possible in C++ but not in C. In this convention:
- Arguments are pushed on the stack from Right to Left.
- The this pointer is passed via register ECX, and not on the stack.
- Since we pass this pointer as well, we cannot use this calling convention for non-member functions.
- Callee cleans the stack.
Let us look at the assembly code for main():
result = obj.thiscallAdd(5, 6);
Ruby
; Line 51 push 6 push 5 lea ecx, DWORD PTR _obj$[ebp] call ?thiscallAdd @Temp @@QAEHHH @Z ; Temp: :thiscallAdd mov DWORD PTR _result$[ebp], eax |
Here, we can see in thiscallAdd, first, the arguments are passed in Right to Left order. Then, the ‘this’ pointer(which refers to the object itself) is passed on to the ECX register.
The function with calling convention __thiscall is thiscalllAdd() inside the Temp class, which is called on line 51 in the main function.
We need to create a class with a member function ‘thiscallAdd’ to use the __thiscall convention. This is because __thiscall can only be used for member functions of a class. In fact, you cannot even use the keyword static when using this calling convention. So, __thiscall can only appear on non-static member functions.
thiscallAdd()
Ruby
?thiscallAdd @Temp @@QAEHHH @Z PROC ; Temp: :thiscallAdd , COMDAT ; _this$ = ecx ; File c:\users\ruchit\documents\code\visual studio\visual studio\source.cpp ; Line 29 push ebp mov ebp, esp sub esp, 216 ; 000000d8H push ebx push esi push edi push ecx lea edi, DWORD PTR [ebp- 216 ] mov ecx, 54 ; 00000036H mov eax, - 858993460 ; ccccccccH rep stosd pop ecx mov DWORD PTR _this$[ebp], ecx mov ecx, OFFSET __F81044A6_source @cpp call @__CheckForDebuggerJustMyCode @4 ; Line 30 mov eax, DWORD PTR _a$[ebp] add eax, DWORD PTR _b$[ebp] mov DWORD PTR _c$[ebp], eax ; Line 31 mov eax, DWORD PTR _c$[ebp] ; Line 32 pop edi pop esi pop ebx add esp, 216 ; 000000d8H cmp ebp, esp call __RTC_CheckEsp mov esp, ebp pop ebp ret 8 ?thiscallAdd @Temp @@QAEHHH @Z ENDP ; Temp: :thiscallAdd |
Explanation: The ‘this’ in classes is a self-reference and thus, the name of the convention: __thiscall.
When we look at the last statement executed, i.e. ret 8, we see that we are adding 8 bytes to the stack pointer and then returning the control to the main. The callee has popped the 2 arguments passed (4 bytes each)and has cleared up the stack.
Advantages of Calling Conventions
The following are the advantages of calling conventions in C/C++:
- The calling convention helps in the standardization of the compiler’s way of function invoking and parameter passing.
- Standardization leads to interoperability between programming languages.
- Calling conventions implements efficient methods to invoke functions according to the requirements.
Contact Us