__thiscall

The __thiscall calling convention is the default calling convention used by methods inside a class. That is why it is only possible in C++ but not in C. In this convention:

• Arguments are pushed on the stack from Right to Left.
• The this pointer is passed via register ECX, and not on the stack.
• Since we pass this pointer as well, we cannot use this calling convention for non-member functions.
• Callee cleans the stack.

Let us look at the assembly code for main():

result = obj.thiscallAdd(5, 6);

Here, we can see in thiscallAdd, first, the arguments are passed in Right to Left order. Then, the ‘this’ pointer(which refers to the object itself) is passed on to the ECX register.

The function with calling convention __thiscall is thiscalllAdd() inside the Temp class, which is called on line 51 in the main function.

We need to create a class with a member function ‘thiscallAdd’ to use the __thiscall convention. This is because __thiscall can only be used for member functions of a class. In fact, you cannot even use the keyword static when using this calling convention. So, __thiscall can only appear on non-static member functions.

thiscallAdd()

Explanation: The ‘this’ in classes is a self-reference and thus, the name of the convention: __thiscall.

When we look at the last statement executed, i.e. ret 8, we see that we are adding 8 bytes to the stack pointer and then returning the control to the main. The callee has popped the 2 arguments passed (4 bytes each)and has cleared up the stack.

In C/C++ programming, a calling convention is a set of rules that specify how a function will be called. You might have seen keywords like __cdecl or __stdcall when you get linking errors. For example:

error LNK2019: unresolved external symbol "void __cdecl A(void)" (?A@@YAXXZ) referenced in function _main

Here, we see __cdecl being used. It is one of the calling conventions in C/C++. You can also see code like this in 3rd party libraries.

For example:

extern __m128i __cdecl _mm256_mask_cvtepi32_epi16(__m128i, __mmask8, __m256i);

What are caller and callee?

When we call a function, a stack frame is allocated for that function, arguments are passed to the function, and after the function does its work, the allocated stack frame is deallocated and control is passed to the calling function.

The function that calls the subroutine is called the caller. The function that gets called(i.e. subroutine) by the caller is called the callee.

In the above code, main() is the caller and func() is the callee.