Detection of stack smashing [Professional Edition only]

4.2.4.12 Detection of stack smashing [Professional Edition only]

The function for detecting stack smashing is implemented by the -Xstack_protector option, -Xstack_protector_all option, or the #pragma directives described in this section.

#pragma stack_protector [(]function-name[(num=value)][)]

#pragma no_stack_protector [(]function-name[)]

-	Generates a code for detection of stack smashing at the entry and end of a function. A code for detection of stack smashing indicates the instructions for executing the three processes shown below.

(1) A 4-byte area is allocated just before the local variable area (in the direction towards address 0xFFFFFFFF) at the entry to a function, and the value specified by <number> is stored in the allocated area.

(2) At the end of the function, whether the 4-byte area in which <number> was stored has been rewritten is checked.

(3) If the value has been rewritten in (2), the __stack_chk_fail function is called as the stack has been smashed.

-	A decimal number from 0 to 4294967295 should be specified in <number>. If the specification of <number> is omitted, the compiler automatically specifies the number.

-	The __stack_chk_fail function needs to be defined by the user and the processing to be executed upon detection of stack smashing should be written. Note the following items when defining the __stack_chk_fail function.

-	The only possible type of return value is void and the __stack_chk_fail function does not have formal parameters.

-	Do not define the function as static.

-	It is prohibited to call the __stack_chk_fail function as a normal function.

-	The __stack_chk_fail function does not generate a code for detection of stack smashing regardless of the -Xstack_protector and -Xstack_protector_all options, and #pragma stack_protector.

-	Prevent returning to the caller, that is, the function where stack smashing was detected by taking measures such as calling abort() in the __stack_chk_fail function to terminate the program.

-	A code for detection of stack smashing is not generated for a function for which #pragma no_stack_protector has been specified regardless of the -Xstack_protector option and -Xstack_protector_all option.

-	If this option is used simultaneously with #pragma stack_protector, the -Xstack_protector option, or the -Xstack_protector_all option, the specification by #pragma becomes valid.

-	An error will occur when #pragma stack_protector and #pragma no_stack_protector are specified simultaneously for the same function within a single translation unit.

-	An error will occur if the specified function is specified for any of the following functions.

#pragma inline, inline keyword

#pragma inline_asm

Example

-	<C source>

#include <stdio.h>

#include <stdlib.h>

#pragma stack_protector f1(num=1234)

void f1()   // Sample program in which the stack is smashed

    volatile char str[10];

    int i;

    for (i = 0; i <= 10; i++){

      str[i] = i; // Stack is smashed when i=10

void __stack_chk_fail(void)

    printf("stack is broken!");

    abort();

-	<Output code>

_f1:

        .stack _f1 = 16

        add 0xFFFFFFF0, r3

        movea 0x000004D2, r0, r1  ; The specified <number> 1234 is stored in the

                                  ; stack area.

        st.w r1, 0x0000000C[r3]

        mov 0x00000000, r2

        br9 .BB.LABEL.1_2

.BB.LABEL.1_1: ; bb

        movea 0x00000002, r3, r5

        add r2, r5

        st.b r2, 0x00000000[r5]

        add 0x00000001, r2

.BB.LABEL.1_2: ; bb7

        cmp 0x0000000B, r2

        blt9 .BB.LABEL.1_1

.BB.LABEL.1_3: ; return

        ld.w 0x0000000C[r3], r1   ; Data is loaded from the location where <number>

        movea 0x000004D2, r0, r12 ; was stored at the entry to a function and

        cmp r12, r1               ; it is compared with the specified num 1234.

        bnz9 .BB.LABEL.1_5        ; If they do not match, a branch occurs.

.BB.LABEL.1_4: ; return

        dispose 0x00000010, 0x00000000, [r31]

.BB.LABEL.1_5: ; return

        br9 ___stack_chk_fail     ; __stack_chk_fail is called.

___stack_chk_fail:

        .stack ___stack_chk_fail = 4

        prepare 0x00000001, 0x00000000

        mov #.STR.1, r6

        jarl _printf, r31

        jarl _abort, r31

        dispose 0x00000000, 0x00000001, [r31]