I have been building my own hypervisor for a few weeks now and part of that process is running real software against it to see what happens and at some point one of the programs I ran caused a "VM entry failure due to invalid guest state". After digging into what was actually triggering it I ended up finding an edge case in how a lot of hypervisors handle VM exits in 32 bit execution mode. What started as debugging my own implementation turned into a viable hypervisor detection technique that works in a lot of open source hypervisors as a lot of them did not think about this scenario.

This post walks through why and how the detection works, a implementation of the detection, and how to fix your implementation.

Detection

The detection relies on a specific interaction between VM exits and 32 bit execution mode that a lot of hypervisor implementations do not account for. In 32 bit mode the highest address the instruction pointer can hold is 0xFFFFFFFF and if execution reaches the end of that address space the CPU naturally wraps the IP back to 0x0. In native mode this is always handled by the CPU but in VMX operation there are scenarios where the hypervisor has to handle the logic behind correctly incrementing the IP.

CPUID is a 2 byte instruction that unconditionally causes a VM exit. If you place a CPUID instruction at 0xFFFFFFFE and jump to it in 32 bit mode a VM exit is caused and execution is redirected to your VM exit handler in root operation. Hypervisors commonly tend to handle the instruction pointer update after the VM exit similarly to the following:

ctx->rip += read_vmcs_field(VMEXIT_INSTRUCTION_LENGTH);

This implementation usually works but in this situation CPUID sits at 0xFFFFFFFE and is 2 bytes long so this calculation tries to set RIP to 0x100000000 which is above the 32 bit address limit. Unlike in non-vmx operation, your hypervisor is not going to automatically wrap that value back to 0x0 so you write an invalid address into the guest RIP, the VM attempts to resume, and you immediately get a VM entry failure due to invalid guest state. This repeats every time the hypervisor tries to resume the guest creating a crash loop which will usually require the user to reboot their computer to get out of the crash loop.

Any usermode program is able to implement this and below is a partial implementation I have made of this detection.

Implementation

// detection.c

extern BOOL trigger32BitJmp(PVOID, PVOID);

LONG filter(struct _EXCEPTION_POINTERS* pInfo) {
    PEXCEPTION_RECORD pRecord = pInfo->ExceptionRecord;
    PCONTEXT pContext = pInfo->ContextRecord;
    if (pRecord->ExceptionCode == STATUS_ACCESS_VIOLATION && pRecord->ExceptionAddress == 0 && pContext->SegCs == 0x23) {
        // We save values in trigger32BitJmp and we restore them here
        pContext->SegCs = pContext->Rdi;
        pContext->Rip = pContext->R10;
        // Set this to vm not detected
        pContext->R15 = 0;
        return EXCEPTION_CONTINUE_EXECUTION;
    }

    return EXCEPTION_CONTINUE_SEARCH;
}

BOOL detectVM() {
    HANDLE SectionHandle;
    LARGE_INTEGER MaximumSize;
    MaximumSize.QuadPart = 0x30000;
    NTSTATUS result = NtCreateSection(&SectionHandle, SECTION_ALL_ACCESS, NULL, &MaximumSize, PAGE_EXECUTE_READWRITE, SEC_COMMIT, NULL);
    if (result != STATUS_SUCCESS) {
        return 1; // Actually implement real error handling lmao
    }

    SIZE_T ViewSize = 0;
    // BaseAddress will always go upto 0xFFFFFFFF because we are subtracting the segment size from 0x100000000
    PVOID BaseAddress = (PVOID)(0x100000000-MaximumSize.QuadPart);
    result = NtMapViewOfSection(SectionHandle, GetCurrentProcess(), &BaseAddress, 0, MaximumSize.QuadPart, NULL, &ViewSize, ViewUnmap, 0, PAGE_EXECUTE_READWRITE);
    if (result != STATUS_SUCCESS) {
        NtClose(SectionHandle);
        return 1; // Actually implement real error handling lmao
    }
	
    // 0x23 will be CS Segment Selector which is the 32 bit compatibility code segment on windows
    // 0xFFFFFFFE is what we will jump to in the far jump
    *((uint64_t*)((uintptr_t)(BaseAddress) + MaximumSize.QuadPart - 0x10)) = 0x23FFFFFFFE;
    // 0F A2 is the opcode for cpuid in x86
    *((unsigned short*)((uintptr_t)(BaseAddress) + MaximumSize.QuadPart - 0x2)) = 0xA20F;

    // We will need a VEH or SEH to catch the exception when it wraps around to 0x0 which will be a access violation
    PVOID vehHandle = AddVectoredExceptionHandler(1, filter);

    BOOL detected = trigger32BitJmp(BaseAddress, (PVOID)((uintptr_t)(BaseAddress) + MaximumSize.QuadPart - 0x10));

    RemoveVectoredExceptionHandler(vehHandle);
    NtUnmapViewOfSection(GetCurrentProcess(), BaseAddress);
    NtClose(SectionHandle);

    return detected;
}

; detection.asm

.code
detection proc
push rbx
push r15
push rbp
push rsi
push rdi

mov r15, 1h ; our detection value if its 1 then there is a vm its set to 0 in the exception filter
mov r8, gs:[8h] ; save stack base
mov r9, gs:[10h] ; save stack limit
lea r10, [return_from_32] ; save RIP we will jump to after the exception filter
mov ax, cs
movzx rdi, ax ; save the old cs segment selector
mov rsi, rsp
mov gs:[8h], rdx ; prevent the stack from overriding what we wrote in 0xFFFFFFF0
mov gs:[10h], rcx ; set our segment as the stack so we can return when we call
lea esp, [edx-100h] ; set the stack to a little before our stuff so we can save the return address
call jump_to_32
jmp real_return

jump_to_32:
jmp fword ptr [edx] ; our far jump into 32 bit mode which is when cpuid is ran
return_from_32:
ret

real_return:
mov rsp, rsi
mov gs:[8h], r8
mov gs:[10h], r9
mov rax, r15
pop rbx
pop rdi
pop rsi
pop rbp
pop r15
ret
detection endp
end

Fix

The fix is pretty simple after seeing why the crash happens. After adding the instruction length to the RIP you can check the value of the long bit in the guest’s code segment access rights which should be 0 if the guest is in 32 bit mode. If the long bit is 0 then you should just use a bitwise AND the rip and 0xFFFFFFFF to remove all bits outside of the 32 bit address range.

uint64_t og_rip = ctx->rip;
ctx->rip += read_vmcs_field(VMEXIT_INSTRUCTION_LENGTH);
if (op_rip < 0x100000000 && ctx->rip >= 0x100000000) {
	SegmentAccessRights cs_access_rights;
	cs_access_rights.raw = read_vmcs_field(GUEST_CS_ACCESS_RIGHTS);
	if (!cs_access_rights.long_mode) {
		ctx->rip &= 0xFFFFFFFF
	}
}
// ... write ctx->rip to vmcs guest rip and whatever

Conclusion

This was a interesting detection I discovered when just testing my hypervisor with a few programs which Roblox’s Hyperion is what caused this detection to happen so credit to them. The detection is not very complex as it is just an edge case that almost nobody tests for because normally you will never get anywhere near the top of the 32 bit address space in a 32 bit program.

Most of the big hypervisors like VMWare and KVM have already fixed this problem but a lot of smaller hypervisors are still affected by this problem.

This is my first blog post I have written so if there is any anything thats weird or wrong you can contact me with the contacts in the about me page :).