Click here to Skip to main content
15,868,419 members
Articles / Programming Languages / C++/CLI
Article

The virtual bool bug

Rate me:
Please Sign up or sign in to vote.
4.88/5 (34 votes)
2 Sep 20034 min read 168.6K   244   27   33
Describes the virtual bool bug that exists in mixed mode Managed C++ programs that access unmanaged classes

Introduction

This bug was first reported by Jochen Kalmbach on April 12th 2002 (no links available to original posting), when VS.NET 7.0 was doing its initial rounds; and it's quite inconceivable why the bug still exists in VS.NET 2003. Just about every week, at least two people report issues related to this bug and I thought it might be a good idea to have an article on it here on CodeProject. What's really annoying is that the developer might spend several hours or even a full day on the problem before realizing that it is not a problem with his code.

The bug

The most common scenario where the bug is reported is when someone has a mixed mode C++ program that has a managed class, which accesses an unmanaged class in an unmanaged DLL. Now if the unmanaged class has a virtual function that returns a bool, then irrespective of what value it returns, the managed caller *always* gets back true. But it's not necessary for the code to be in two separate entities (the EXE and the DLL), the bug occurs if the unmanaged class is defined in a #pragma unmanaged block in a mixed mode EXE or DLL.

Minimal code to reproduce bug

MC++
#pragma unmanaged
class Unmanaged
{
public:
    virtual bool IsAlive()
    {
        return false;
    }
};

#pragma managed
__gc class Managed
{
public:
    void Test()
    {
        Unmanaged* um = new Unmanaged();
        if(um->IsAlive())
        {
            //Always executes
            Console::WriteLine("Function returned true. BUG!!!");
        }
        else
        {
            //Never executes
            Console::WriteLine("Function returned false. No Bug :-)");
        }
        delete um;
    }
};

int _tmain()
{
    Managed* mg = new Managed();
    mg->Test();
    return 0;
}

Trying to figure it out

Let's examine the disassembly for the IsAlive function :-

ASM
;virtual bool IsAlive()
004010B0 push ebp 
004010B1 mov ebp,esp 
004010B3 push ecx 
004010B4 mov dword ptr [ebp-4],ecx 

;return false
004010B7 xor al,al ; Notice how AL is made 0 (false)
004010B9 mov esp,ebp 
004010BB pop ebp 
004010BC ret 

As you can see, the result of the function is returned in the AL register and this is what the contents of my registers looked like at this point :-

EAX = 00401000 EBX = 0012EFB4 ECX = 06C42C88 EDX = 00425410 
ESI = 00168930 EDI = 00000000 EIP = 004010B9 ESP = 0012EFA8 
EBP = 0012EFAC EFL = 00000246

Now let's see the disassembly for the caller code :-

ASM
;if(um->IsAlive())
00000065 mov eax,dword ptr [ebp-18h] 
00000068 mov eax,dword ptr [eax] 
0000006a mov esi,dword ptr [eax] 
0000006c mov ecx,dword ptr [ebp-18h] 
0000006f mov eax,esi 
00000071 push 1692D0h 
00000076 call F9759F50 ; The call to the function
0000007b movzx esi,al ; Copying the return value to ESI
0000007e test esi,esi ; Checking for true 
00000080 je 0000009A ; If false then jump to 9A  

The return value is obtained from the AL register. Let's see the contents of the registers now :-

EAX = 00000001 EBX = 0012F0C8 ECX = 00000004 EDX = 00000000 
ESI = 00000001 EDI = 04A719C8 EBP = 0012F070 ESP = 0012F044 

Horror of horrors! AL is now 1 (more precisely

MC++
EAX
has been set to 1). I had stepped through the disassembly and
MC++
AL
was 0 at the time the RET instruction was executed; therefore the register corruption must have occurred during the managed-unmanaged transition.

Workarounds

The simple workaround is to use a BOOL (typedef for an int) instead of a bool.

MC++
class Unmanaged
{
public:
    virtual int IsAlive()
    {
        return false;
    }
};

The casting is implicit from am int to a

MC++
bool
and so we don't really have to do anything extra.

A slightly bizarre looking workaround [see section titled "More info" for heheh more info] suggested by someone (possibly Microsoft Support) is to set EAX to a value under 255 before returning from the unmanaged function.

MC++
class Unmanaged
{
public:
    virtual bool IsAlive()
    {
        __asm mov eax,100
        return false;
    }
};

More info

I got some more information regarding this issue from Tom Archer (my friend, fellow-CPian and co-author) who got this information from a friend of his, who is in the VC++ compiler team. It seems this bug occurs when one of the upper 24 bits of the EAX register is non-zero. They have a hot-fix for this bug for both VC++.NET 7 and for VC++.NET Everett, but it might be a better idea to wait for the next service pack.

Still more info (Thanks Jochen)

Jochen's post gave me a few links which provided even more info on this bug. The bug occurs due to the way the CLR marshals boolean values. The CLR thinks that a boolean is 4 bytes (as it is under .NET) but the C++ bool type is only a single byte (so much for efficiency and the hassles it brings about). What happens during marshalling is that the CLR examines the higher three bytes and if they contain any data, it assumes that the boolean value being passed is true. As far as I understood from the postings made by MS support, there was a sort of vague argument between the CLR team and the VC++ compiler team. The VC++ compiler team believed (and rightly so in my opinion) that the issue was with the CLR's marshalling code, but it seems the CLR team wanted the VC++ team to emit a custom MarshalAs attribute for the method that returns a bool. But obviously you cannot apply .NET attributes to an unmanaged function, as methods compiled as unmanaged don't appear in the meta-data. Anyway, now we know why its so important to clear the upper 3 bytes of the EAX register.

Related Microsoft KB links

Conclusion

What's really dangerous about this bug is that it's quite easy not to see it, because most functions that return bool return

MC++
false
to indicate an error, and thus by getting true all the time, we never realize that there is anything amiss. Thus it's quite easy to miss the bug until it's really late into the software development cycle. I have been working with mixed mode programs for quite a while now, specially since I began my book with Tom (Extending MFC Applications With the .NET Framework) and this is an issue in which I am quite naturally interested; and I would like to hear more intelligent analysis than mine from some of the gurus that frequent CP.

History

  • Aug 27 2003 - First published
  • Aug 30 2003 - Updated with more info and related KB link
  • Sep 03 2003 - Updated with more info provided by Jochen

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here


Written By
United States United States
Nish Nishant is a technology enthusiast from Columbus, Ohio. He has over 20 years of software industry experience in various roles including Chief Technology Officer, Senior Solution Architect, Lead Software Architect, Principal Software Engineer, and Engineering/Architecture Team Leader. Nish is a 14-time recipient of the Microsoft Visual C++ MVP Award.

Nish authored C++/CLI in Action for Manning Publications in 2005, and co-authored Extending MFC Applications with the .NET Framework for Addison Wesley in 2003. In addition, he has over 140 published technology articles on CodeProject.com and another 250+ blog articles on his WordPress blog. Nish is experienced in technology leadership, solution architecture, software architecture, cloud development (AWS and Azure), REST services, software engineering best practices, CI/CD, mentoring, and directing all stages of software development.

Nish's Technology Blog : voidnish.wordpress.com

Comments and Discussions

 
NewsStill broken in Compact Framework 3.5 Pin
btjdev29-Sep-09 7:45
btjdev29-Sep-09 7:45 
GeneralUnmanaged calling Unmanaged Pin
Jim99916-Aug-07 8:22
Jim99916-Aug-07 8:22 
QuestionVisual Studio 2005? Pin
Paul Tulou15-Aug-07 4:15
Paul Tulou15-Aug-07 4:15 
Questionmanaged/unmanaged boundary Pin
ensdorf13-Nov-06 12:34
ensdorf13-Nov-06 12:34 
AnswerRe: managed/unmanaged boundary Pin
Paul Tulou15-Aug-07 4:53
Paul Tulou15-Aug-07 4:53 
GeneralStill fixed in VC2005-Express Beta 1 Pin
Jochen Kalmbach [MVP VC++]29-Jun-04 9:53
Jochen Kalmbach [MVP VC++]29-Jun-04 9:53 
GeneralRe: Still fixed in VC2005-Express Beta 1 Pin
Nish Nishant5-Apr-05 21:22
sitebuilderNish Nishant5-Apr-05 21:22 
QuestionHow did this bug make it into a release? Pin
Anonymous16-Jun-04 16:36
Anonymous16-Jun-04 16:36 
AnswerRe: How did this bug make it into a release? Pin
Nish Nishant5-Apr-05 21:21
sitebuilderNish Nishant5-Apr-05 21:21 
GeneralFixed in Whidbey alpha.... Pin
Jochen Kalmbach [MVP VC++]16-Jan-04 8:00
Jochen Kalmbach [MVP VC++]16-Jan-04 8:00 
GeneralRe: Fixed in Whidbey alpha.... Pin
Nish Nishant5-Apr-05 21:20
sitebuilderNish Nishant5-Apr-05 21:20 
GeneralReminder: Avoid Mixed DLLs Pin
Roy Muller4-Sep-03 5:29
Roy Muller4-Sep-03 5:29 
GeneralRe: Reminder: Avoid Mixed DLLs Pin
Jochen Kalmbach [MVP VC++]5-Sep-03 0:25
Jochen Kalmbach [MVP VC++]5-Sep-03 0:25 
GeneralRe: Reminder: Avoid Mixed DLLs Pin
Roy Muller5-Sep-03 5:07
Roy Muller5-Sep-03 5:07 
GeneralI am not the first... Pin
Jochen Kalmbach [MVP VC++]3-Sep-03 4:55
Jochen Kalmbach [MVP VC++]3-Sep-03 4:55 
GeneralRe: I am not the first... Pin
Nish Nishant3-Sep-03 5:04
sitebuilderNish Nishant3-Sep-03 5:04 
GeneralRe: I am not the first... Pin
Jochen Kalmbach [MVP VC++]3-Sep-03 6:54
Jochen Kalmbach [MVP VC++]3-Sep-03 6:54 
GeneralRe: I am not the first... Pin
Jochen Kalmbach [MVP VC++]3-Sep-03 6:55
Jochen Kalmbach [MVP VC++]3-Sep-03 6:55 
GeneralRe: I am not the first... Pin
Nish Nishant3-Sep-03 15:28
sitebuilderNish Nishant3-Sep-03 15:28 
GeneralRe: I am not the first... Pin
Nish Nishant3-Sep-03 5:06
sitebuilderNish Nishant3-Sep-03 5:06 
GeneralIsnt this the same as.... Pin
leppie30-Aug-03 2:55
leppie30-Aug-03 2:55 
GeneralRe: Isnt this the same as.... Pin
Gary R. Wheeler30-Aug-03 13:11
Gary R. Wheeler30-Aug-03 13:11 
Your two examples don't do the same thing. Example 1 returns a value of type BOOL:
BOOL SomeFunction(...)
    BOOL res = FALSE;
    //...
    return res;
}

whereas example 2, if the function it's defined in is like this:
BOOL SomeFunction(...)
    BOOL res = FALSE;
    //...
    return (res) != 0;
}
is doing something different. The expression (res) != 0 is an expression that evaluates to a bool, which is then promoted to a value of type BOOL before being returned. The promotion will work as long as the BOOL constants of TRUE and FALSE are defined as 1 and 0, respectively.

I believe it's safer to use explicit conversion functions like the following:
inline bool _bool(BOOL value)
{
    return ((value != FALSE) ? true : false);
};
inline BOOL _BOOL(bool value)
{
    return (value ? TRUE : FALSE);
}
The _bool function converts a Windows BOOL value to a C++ bool. Note that this function works even if it is ostensibly 'TRUE', but not equal to TRUE. There are a number of API's that supposedly return a BOOL, that actually return a simple DWORD where zero indicates failure and nonzero indicates success (or vice versa). The _BOOL function is obvious.


Software Zen: delete this;
GeneralA bit more detail Pin
Tom Archer29-Aug-03 11:21
Tom Archer29-Aug-03 11:21 
GeneralRe: A bit more detail Pin
John M. Drescher29-Aug-03 11:40
John M. Drescher29-Aug-03 11:40 
GeneralRe: A bit more detail Pin
Tom Archer29-Aug-03 11:50
Tom Archer29-Aug-03 11:50 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.