This is a summary of a way too long debugging session leading to unexpected places. Hopefully someone might find it useful or interesting. This post also describes a memory leak bug in current (as of October 2024) versions of Windows 10 & 11. Microsoft is well known for having a streamlined process for reporting & fixing bugs in their products ¹, so I'm following the best practices by "Writing an Angry Blog Post".

The problemTop

Here is a very simple program which repeatedly initializes and releases the H.264 decoder shipped with Windows. For some mysterious reason, it leaks a few kilobytes of memory in each iteration.

#include <windows.h>
#include <mfapi.h>
#include <mferror.h>
#include <wmcodecdsp.h>
#include <mfidl.h>
#include <codecapi.h>

int main()
{
        for(;;) {
                CoInitializeEx(nullptr, COINIT_MULTITHREADED | COINIT_DISABLE_OLE1DDE);
                IUnknown* unk= 0;
                CoCreateInstance(CLSID_CMSH264DecoderMFT, NULL, CLSCTX_INPROC_SERVER, IID_IUnknown, (void**)&unk);
                unk->Release();
                CoUninitialize();
        }
        return 0;
}

Any leak tracing tools that I'd tried would say: "nope, no leaks here". VMMap was marginally helpful, letting me know that new pages are appearing in my process out of somewhere. Yet no calls to VirtualAlloc & friends were made ².

Going deeperTop

At this point, I've assumed that there is some other function which allocates memory that I'm not aware of ³. So I've began a long WinDbg session of step-into/step-over while monitoring if the new memory area has been allocated. In most cases, the areas would be allocated sequentially, so I knew where to expect the new memory to pop up. Finally, I've reached the surprising culprit: a call to NtQuerySystemInformation somewhere inside msmpeg2vdec.dll.

Why - on earth - would NtQuerySystemInformation allocate memory?
What does a SystemInformationClass parameter of 185 mean?

The answer to these questions is "Warbird".

Warbird?Top

Warbird is an undocumented code obfuscation/DRM solution from Microsoft, used internally in their products. For simplicity, we will focus on the kernel component ⁴, which is accessible through the NtQuerySystemInformation syscall. See this talk by Alex Ionescu (slides) for a detailed description. Warbird's kernel component supports the following operations (among others):

ProcessStartup
Decrypt
Encrypt
HeapExecuteCall
HeapExecuteReturn
ProcessModuleUnload

The most interesting part is HeapExecuteCall. This operation performs the following sequence:

allocate a temporary memory area in the user process
copy & decrypt the provided memory block into the area
return execution into this decrypted code

HeapExecuteReturn is a counterpart executed from within the decrypted code block - it returns execution flow to the original call site. Note that neither of these calls is responsible for releasing the allocated memory. The upside is that within HeapExecuteCall, the memory is obtained from a "cache", so the memory will be reused for subsequent calls (my understanding is that these calls can be nested, so more than a single block may be needed).

Root causeTop

When msmpeg2vdec.dll is unloaded, it calls the Warbird function ProcessModuleUnload. My guess is that it handles some cleanup steps which are necessary when a Warbird-encumbered library is being unloaded. Notably, it drops the "cache" holding information about the memory blocks mentioned earlier.

BUT THE BLOCKS THEMSELVES ARE LEAKED

This is why the sample at the beginning of this post needs to have CoInitialize/CoUninitialize inside the loop. Otherwise the library is not reloaded and the leak does not occur.

SolutionTop

Not really. It took MS several months to fix a notable security issue within Warbird. I'm not expecting them to fix a memory leak which may happen only under very specific conditions.

If you refrain from repeatedly loading & unloading Warbird-infested libraries, the memory will not be leaked.

Footnotes

1 Ha, ha!

2 There was one call to VirtualAlloc, but it was properly released in each loop, so it doesn't count.

3 In hindsight, there could be another possibility - an external process could be allocating pages on our behalf using VirtualAllocEx.

4 I'm sure that someone at MS had the realization that DRM is not a shitty enough in itself, so it needs to be placed into the kernel to ensure maximum nuisance levels.

Add comment