Bug #11466: Memory leak in win32 fill_random_bytes_syscall

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: master
• Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN, 2.2: UNKNOWN

[scorpion007 (Alex Budovski)]

Leak of 608 bytes in this stack:

+     608 (    608 -      0)      1 allocs	BackTrace9BD0704B
+       1 (      1 -      0)	BackTrace9BD0704B	allocations

	ntdll!RtlpCallInterceptRoutine+40 (d:\blue\minkernel\ntos\rtl\heappriv.h, 3625)
	ntdll!RtlAllocateHeap+79836 (d:\blue\minkernel\ntos\rtl\heap.c, 1892)
	rsaenh!InitUser+20 (d:\9147\ds\win32\ntcrypto\scp\ntagum.c, 783)
	rsaenh!NTagLogonUser+167 (d:\9147\ds\win32\ntcrypto\scp\ntagum.c, 950)
	rsaenh!CPAcquireContext+88 (d:\9147\ds\win32\ntcrypto\scp\ntagum.c, 1230)
	cryptsp!CryptAcquireContextA+4A3 (d:\9147\ds\win32\ntcrypto\cryptsp\cryptapi.c, 884)
	rubyprov!fill_random_bytes_syscall+44 (e:\dev\ruby\random.c, 498)
	rubyprov!fill_random_bytes+1D (e:\dev\ruby\random.c, 542)
	rubyprov!fill_random_seed+2A (e:\dev\ruby\random.c, 557)
	rubyprov!init_randomseed+18 (e:\dev\ruby\random.c, 1446)
	rubyprov!Init_RandomSeed+3C (e:\dev\ruby\random.c, 1458)
	rubyprov!rb_call_inits+E (e:\dev\ruby\inits.c, 21)
	rubyprov!ruby_setup+C6 (e:\dev\ruby\eval.c, 63)

It looks like the fill_random_bytes_syscall isn't releasing the context on success?

static int
fill_random_bytes_syscall(void *seed, size_t size)
{
    static HCRYPTPROV perm_prov;
    HCRYPTPROV prov = perm_prov, old_prov;
    if (!prov) {
	if (!CryptAcquireContext(&prov, NULL, NULL, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)) {
	    prov = (HCRYPTPROV)INVALID_HANDLE_VALUE;
	}
	old_prov = (HCRYPTPROV)ATOMIC_PTR_CAS(perm_prov, 0, prov);
	if (LIKELY(!old_prov)) { /* no other threads acquried */
	    if (prov != (HCRYPTPROV)INVALID_HANDLE_VALUE) {
		rb_gc_register_mark_object(Data_Wrap_Struct(0, 0, release_crypt, &perm_prov));
	    }
	}
	else {			/* another thread acquried */
	    if (prov != (HCRYPTPROV)INVALID_HANDLE_VALUE) {
		CryptReleaseContext(prov, 0);
	    }
	    prov = old_prov;
	}
    }
    if (prov == (HCRYPTPROV)INVALID_HANDLE_VALUE) return -1;
    CryptGenRandom(prov, size, seed);
    return 0;
}

[jeremyevans0 (Jeremy Evans)]

• Status changed from Open to Closed

If fill_random_bytes_syscall were to leak memory, then this Ruby code should show the leak:

i = 0; while true; puts i if (i+=1) % 10000000; SecureRandom.bytes(1) end

As this shows no leak even with with 600000000 iterations, I am going to guess the leak checker used is flawed, or maybe is a one-time leak of 608 bytes when the function is first called, not a leak per call, in which case it can be safely ignored.