Compiler Security Checks In Depth

Contents

Introduction
What Is a Buffer Overrun?
Anatomy of the x86 Stack
Run-Time Checks
What /GS Does
The Error Handler
The Cookie Value
Performance Impact
Examples
Conclusion

Introduction

Software security is a major concern for the high-tech industry, and the most feared and misunderstood software vulnerability is the buffer overrun. Today, the mention of a buffer overrun is enough to make people stop and listen. All too often, the technical details get lost in the transcription, and the general public comes away with a rather alarming view of a rather fundamental problem. To address this problem, Visual C++ .NET introduces security checks to assist developers in identifying buffer overruns.

What Is a Buffer Overrun?

Buffers are blocks of memory, usually in the form of an array. When the size of an array is not verified, it is possible to write outside the allocated buffer. If such an action takes place in memory addresses higher than the buffer, it is called a buffer overrun. A similar problem exists when writing to a buffer in memory addresses below the allocated buffer. In this case, it is called a buffer underflow. Underflows are significantly rarer than overruns, but they do show up, as described later in this article.

A certain class of well-documented functions, including strcpy, gets, scanf, sprintf, strcat, and so on, is innately vulnerable to buffer overruns, and their use is actively discouraged. A simple example shows the danger of such functions:

int vulnerable1(char * pStr) {
    int nCount = 0;
    char pBuff[_MAX_PATH];

    strcpy(pBuff, pStr);

    for(; pBuff; pBuff++)
       if (*pBuff == '\\') nCount++;

    return nCount;
}

This code has an obvious vulnerability — the pBuff parameter could be overrun if the buffer pointed to by pStr is longer than _MAX_PATH. A simple inclusion of

assert(strlen(pStr)
< _MAX_PATH)

is enough to catch this fallacy at run time for a debug build, but not for a release build. Using these vulnerable functions is considered bad practice if trying to completely avoid such issues. Similar functions that are technically less vulnerable, such as strncpy, strncat, and memcpy, do exist. The problem with these functions is that it is the developer asserting the size of the buffer, not the compiler. An incredibly common mistake is demonstrated in the following function:

#define BUFLEN 16

void vulnerable2(void) {
    wchar_t buf[BUFLEN];
    int ret;

    ret = MultiByteToWideChar(CP_ACP, 0, "1234567890123456789", -1, buf, sizeof(buf));
    printf("%d\n", ret);
}

In this case, the number of bytes instead of the number of characters was used to declare the size of the buffer, and an overflow takes place. To fix this vulnerability, the last argument to MultiByteToWideChar should be sizeof(buf)/sizeof(buf[0]). Both vulnerable1 and vulnerable2 are common mistakes that can easily be prevented; however, if missed by a code review, potentially dangerous security vulnerabilities can be shipped in an application. This is why Visual C++ .NET introduces security checks, which would prevent buffer overruns in both vulnerable1 and vulnerable2 from injecting malicious code into the vulnerable application. A buffer overrun which injects code into a running process is referred to as an exploitable buffer overrun.

Anatomy of the x86 Stack

To fully understand how the environment in which a buffer overrun can be exploited and how security checks work, the layout of the stack must be fully understood. On the x86 architecture, stacks grow downward, meaning that newer data will be allocated at addresses less than elements pushed onto the stack earlier. Each function call creates a new stack frame with the following layout, note that high memory is at the top of the list:

Function parameters
Function return address
Frame pointer
Exception Handler frame
Locally declared variables and buffers
Callee save registers

From the layout, it is clear that a buffer overflow has the opportunity to overwrite other variables allocated before the buffer, the exception frame, the frame pointer, the return address, and the function parameters. To take over the program’s execution, a value must be written in to data that will later be loaded into the EIP register. The function’s return address is one of these values. A classic buffer overrun exploit will overrun the return address and then let the function’s return instruction load the return address into EIP.

The data elements are stored on the stack in the following way. The function parameters are pushed on the stack before the function is called. The parameters are pushed from right to left. The function return address is placed on the stack by the x86 CALL instruction, which stores the current value of the EIP register. The frame pointer is the previous value of the EBP register and is placed on the stack when the frame pointer omission (FPO) optimization does not take place. Therefore, the frame pointer is not always placed in a stack frame. If a function includes try/catch or any other exception handling construct, the compiler will include exception handling information on the stack. Following that, the locally declared variables and buffers are allocated. The order of these allocations can change depending on which optimizations take place. Finally, the callee save registers such as ESI, EDI, and EBX are stored if they are used at any point during the functions execution.

Run-Time Checks

Buffer overruns are common mistakes made by C or C++ programmers, and potentially, and that is notably one of the most dangerous. Visual C++ .NET provides tools that can make it easier for developers to find these errors during the development cycle so that the errors can be fixed. The /GZ switch, found in Visual C++ 6.0, has found new life in the /RTC1 switch of Visual C++ .NET. The /RTC1 switch is an alias for /RTCsu, where s stands for stack checks (the focus of this discussion), and the u stands for uninitialized variable checks. All buffers allocated on the stack are tagged at the edges; therefore, overruns and underflows can be caught. While small overruns may not change the execution of the program, they can corrupt data near the buffer, which can go unnoticed.

The run-time checks are incredibly useful to developers who not only want to write secure code but also care about the fundamental issue of writing correct code. The run-time checks only work for debug builds, however; this feature was never designed to operate in production code. Nevertheless, there is an obvious value in doing checks for buffer overflows in production code. Doing so would require a design that has a much smaller performance impact than the run-time check implementation. To that end, the Visual C++ .NET compiler introduces the /GS switch.

What /GS Does

The /GS switch provides a "speed bump," or cookie, between the buffer and the return address. If an overflow writes over the return address, it will have to overwrite the cookie put in between it and the buffer, resulting in a new stack layout:

Function parameters
Function return address
Frame pointer
Cookie
Exception Handler frame
Locally declared variables and buffers
Callee save registers

The cookie will be examined in more detail later. The function's execution does change with these security checks. First, when a function is called, the first instructions to execute are in the function’s prolog. At a minimum, a prolog allocates space for the local variables on the stack, such as the following instruction:

sub   esp,20h

This instruction sets aside 32 bytes for use by local variables in the function. When the function is compiled with /GS, the functions prolog will set aside an additional four bytes and add three more instructions as follows:

sub   esp,24h
mov   eax,dword ptr [___security_cookie (408040h)]
xor   eax,dword ptr [esp+24h]
mov   dword ptr [esp+20h],eax

The prolog contains an instruction that fetches a copy of the cookie, followed by an instruction that does a logical xor of the cookie and the return address, and then finally an instruction that stores the cookie on the stack directly below the return address. From this point forward, the function will execute as it does normally. When a function returns, the last thing to execute is the function’s epilog, which is the opposite of the prolog. Without security checks, it will reclaim the stack space and return, such as the following instructions:

add   esp,20h
ret

When compiled with /GS, the security checks are also placed in the epilog:

mov   ecx,dword ptr [esp+20h]
xor   ecx,dword ptr [esp+24h]
add   esp,24h
jmp   __security_check_cookie (4010B2h)

The stack's copy of the cookie is retrieved and then follows with the XOR instruction with the return address. The ECX register should contain a value that matches the original cookie stored in the __security_cookie variable. The stack space is then reclaimed, and then, instead of executing the RET instruction, the JMP instruction to the __security_check_cookie routine is executed.

The __security_check_cookie routine is straightforward: if the cookie was unchanged, it executes the RET instruction and ends the function call. If the cookie fails to match, the routine calls report_failure. The report_failure function then calls __security_error_handler(_SECERR_BUFFER_OVERRUN, NULL). Both functions are defined in the seccook.c file of the C run-time (CRT) source files.

The Error Handler

CRT support is needed to make these security checks work. When a security check failure occurs, control of the program is passed to __security_error_handler, which is summarized here:

void __cdecl __security_error_handler(int code, void *data)
{
    if (user_handler != NULL) {
      __try {
        user_handler(code, data);
      } __except (EXCEPTION_EXECUTE_HANDLER) {}
    } else {
      //...prepare outmsg...
      __crtMessageBoxA(
          outmsg,
          "Microsoft Visual C++ Runtime Library",
          MB_OK|MB_ICONHAND|MB_SETFOREGROUND|MB_TASKMODAL);
    }
    _exit(3);
}

By default, an application that fails a security check displays a dialog that states “Buffer overrun detected!”. When the dialog is dismissed, the application terminates. The CRT library offers the developer an option to use a different handler that would react to the buffer overrun in a manner more sensible to the application. The function, __set_security_error_handler, is used to install the user handler by storing the user-defined handler in the user_handler variable, as shown in the following example:

void __cdecl report_failure(int code, void * unused)
{
    if (code == _SECERR_BUFFER_OVERRUN)
      printf("Buffer overrun detected!\n");
}

void main()
{
    _set_security_error_handler(report_failure);
    // More code follows
}

A detected buffer overrun in this application will print a message to the console window instead of presenting a dialog box. Although the user handler does not explicitly terminate the program, when the user handler returns, __security_error_handler will terminate the program with a call to _exit(3). Both the __security_error_handler and _set_security_error_handler functions are located in the secfail.c file of the CRT source files.

It is useful to discuss what should be done in the user handler. A common reaction would be to throw an exception. Because exception information is stored on the stack, however, throwing an exception can pass control to a corrupted exception frame. To prevent this, the __security_error_handler function wraps the call to the user function in a __try/__except block that captures all exceptions, followed by termination of the program. The developer does not want to call DebugBreak, because it raises an exception, or use longjmp. All that the user handler should do is report the error and possibly create a log so the buffer overrun can be fixed.

Sometimes, a developer may want to rewrite __security_error_handler rather than use _set_security_error_handler to achieve the same goal. Rewriting is prone to error, and the main handler is important enough that implementing it incorrectly can have devastating results.

The Cookie Value

The cookie is a random value with the same size as a pointer, meaning that, on the x86 architecture, the cookie is four bytes long. The value is stored in the __security_cookie variable with other CRT global data. The value is initialized to a random value by a call to __security_init_cookie found in the seccinit.c file of the CRT source files. The randomness of the cookie comes from the processor counters. Each image (that is, each DLL or EXE compiled with /GS) has a separate cookie value at load time.

Two problems can occur when applications start building with the /GS compiler switch. First, applications that do not include CRT support will lack a random cookie, because the call to __security_init_cookie takes place during CRT initialization. If the cookie is not randomly set at load time, the application remains vulnerable to attack if a buffer overflow is discovered. To solve this problem, the application needs to explicitly call __security_init_cookie during startup. Second, old applications that call the documented _CRT_INIT function to initialize may run into unexpected security check failures, such as in the following example:

DllEntryPoint(...) {
    char buf[_MAX_PATH];   // a buffer that triggers security checks
    ...
    _CRT_INIT();
    ...

}

The problem is that the call to _CRT_INIT changes the value of the cookie while a function already set up for a security check is live. Because the value of the cookie will be different at the function’s exit, the security check interprets that there was a buffer overrun. The solution is to avoid declaring buffers in live functions before the call to _CRT_INIT. Currently, there is a workaround using the _alloca function to allocate the buffer on the stack, because the compiler will not generate a security check if the allocation is done with _alloca. This workaround is not guaranteed to work in future versions of Visual C++.

Performance Impact

A performance tradeoff for using security checks in an application must be made. The Visual C++ compiler team focused on making the performance degradation small. In most cases, the performance should not degrade more than 2 percent. In fact, experience has shown that most applications, including high-performance server applications, have not noticed any performance impact.

The most important factor behind keeping the performance impact from being an issue is that only functions that are vulnerable to attack are targeted. Currently, the definition of a vulnerable function is one that allocates a type of string buffer on the stack. A string buffer that is considered vulnerable allocates more than four bytes of storage and where each element of the buffer is either one or two bytes. Small buffers are unlikely to be the target of an attack, and limiting the number of functions that have security checks limits the code growth. Most executables will not even notice an increase in size when building with /GS.

Examples

Therefore, the /GS switch does not fix buffer overruns, but it can prevent buffer overruns from being exploited in certain situations. vulnerable1 and vulnerable2 are immune from exploitation when compiled with the /GS switch. Any function where a buffer overrun is the last action to occur before returning will be immune from exploitation. Because a buffer overrun can take place early in the execution of a function, circumstances exist where a security check will either not have the opportunity to detect the buffer overrun or the security check may itself have been attacked by the overrun, as shown in the following examples.

Example 1

class Vulnerable3 {
public:
    int value;

    Vulnerable3() { value = 0; }
    virtual ~Vulnerable3() { value = -1; }
};

void vulnerable3(char * pStr) {
    Vulnerable3 * vuln = new Vulnerable3;
    char buf[20];

    strcpy(buf, pStr);
    delete vuln;
}

In this situation, an pointer to an object with virtual functions is allocated on the stack. Because the object has virtual functions, the object includes a vtable pointer. An attacker could seize this opportunity by supplying a malicious pStr value and overrun buf. Before the function returns, the delete operator calls the virtual destructor for vuln. To do this requires looking up the destructor function in the vtable, which has now been taken over. The execution of the program is taken before the function returns, so the security checks were never allowed to detect the buffer overrun.

Example 2

void vulnerable4(char *bBuff, in cbBuff) {
    char bName[128];
    void (*func)() = foo;

    memcpy(bName, bBuff, cbBuff);
    (func)();
}

In this case, the function is vulnerable to a pointer subterfuge attack. When the compiler allocates space for the two locals, it will put the func variable before pName. This is because the optimizer can improve the code’s efficiency with this layout. Unfortunately, this allows an attacker to supply a malicious value for bBuff. Also, an attacker can supply the value of cbBuff, which indicates the size of bBuff. The function mistakenly omits the verification that cbBuff is less than or equal to 128. The call to memcpy can therefore overrun the buffer and trash the value of func. Because the func pointer subterfuge is used to call the function it points to before the vulnerable4 function returns, the hijack takes place before the security check could take place.

Example 3

int vulnerable5(char * pStr) {
    char buf[32];
    char * volatile pch = pStr;

    strcpy(buf, pStr);
    return *pch == '\0';
}

int main(int argc, char* argv[]) {
    __try { vulnerable5(argv[1]); }
    __except(2) { return 1; }
    return 0;
}

This program shows a particularly difficult problem because it uses structured exception handling. As previously mentioned, functions that use exception handling put information, such as the appropriate exception handling functions, on the stack. In this case, the exception handling frame in the main function can be attacked even though vulnerable5 has the flaw. An attacker will take advantage of the opportunity to overrun buf and trash both pch and the exception handling frame of main. Because the vulnerable5 function later dereferences pch, if the attacker supplied a value such as zero, he or she could cause an access violation that in turn raises an exception. During stack unwinding, the operating system looks to the exception frames for exception handlers to which it should pass control. Because the exception handling frame was corrupted, the operating system passes control of the program to arbitrary code supplied by the attacker. The security checks were not able to detect this buffer overrun because the function did not return properly.

Some of the most popular exploits recently have used exception handling exploits. One of the most newsworthy was the Code Red virus that appeared during summer 2001. Windows XP already creates an environment where exception handling attacks are more difficult because the address of the exception handler cannot be on the stack, and all the registers are zeroed out before the exception handler is called.

Example 4

void vulnerable6(char * pStr) {
    char buf[_MAX_PATH];
    int * pNum;

    strcpy(buf, pStr);
    sscanf(buf, "%d", pNum);
}

This function when compiled with /GS, unlike the previous three examples, cannot take over execution by simply overrunning the buffer. This function requires a two-stage attack to take over the program’s execution. Knowing that pNum will be allocated before buf makes it susceptible to being overwritten with an arbitrary value supplied by the pStr string. An attacker would have to choose four bytes of memory to overwrite. If the buffer overwrote the cookie, an opportunity lies in taking over the user handler function pointer stored in the user_handler variable or the value stored in the __security_cookie variable. If the cookie is not overwritten, the attacker would choose the address of the return address for a function that does not contain security checks. In this case, the program would execute as normal, returning from functions unaware of the buffer overrun; a short time later, the program is silently taken over.

Vulnerable code could also be subject to additional attacks, such as a buffer overflow in the heap, which is not addressed by /GS. Index out-of-range attacks that target specific indexes into an array rather than writing to the array sequentially are also not addressed by /GS. An unchecked out-of-range index can essentially target any part of memory and can avoid overwriting the cookie. Another form of unchecked index is a signed/unsigned integer mismatch where a negative number was supplied to an array index. Simply verifying that the index is less than the size of the array is not enough if the index is a signed integer. Lastly, the /GS security checks do not address buffer underflows in general.

Conclusion

Buffer overruns can clearly be a devastating flaw in an application. Nothing replaces writing tight, secure code in the first place. Despite popular opinion, some buffer overruns are extremely difficult to discover. The /GS switch is a useful tool for developers interested in writing secure code. It does not solve the problem that buffer overruns exist in the code, however. Even with the security checks to prevent the buffer overrun from being exploited in some cases, the program still terminates, which is a denial of service attack, especially against server code. Building with /GS is a safe and secure way for developers to mitigate the risks of vulnerable buffers of which he or she was not aware.

While tools exist to flag possible vulnerabilities such as the ones discussed in this article, they are provably imperfect. Nothing compares to a good code review by developers who know what to look for. The book Writing Secure Code by Michael Howard and David LeBlanc provides an excellent discussion of many other ways to mitigate risk when writing highly secure applications.

 

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

© 2002 Microsoft Corporation. All rights reserved.