Documented Behavior

The IO_IGNORE_SHARE_ACCESS_CHECK flag provides a way for Windows kernel components to bypass sharing checks while opening local files. NTFSD has lots of posts on the subject — see here, here, here, etc…

MSDN discusses the flag as well but the wording is somewhat ambiguous. For example, the documentation for FltCreateFileEx2 mentions IO_IGNORE_SHARE_ACCESS_CHECK in a couple of places:

Options flags Meaning
IO_IGNORE_SHARE_ACCESS_CHECK Indicates that the I/O manager should not perform share-access checks on the file object after it is created. However, the file system might still perform these checks.

And later, a similarly ambiguous note:

Note If IO_IGNORE_SHARE_ACCESS_CHECK is specified in the Flags parameter, the I/O manager ignores the ShareAccess parameter. However, the file system might still perform access checks. Thus, it is important to specify the sharing mode you would like for the ShareAccess parameter, even when using the IO_IGNORE_SHARE_ACCESS_CHECK flag.

I’ve been using this flag for years, and I thought I understood the ambiguity in the documentation.

  1. The documentation fails to mention that this flag has no effect on files opened across the network.
  2. My understanding was that this flag *might* not be honored on third-party file systems since they may perform sharing access checks without the help of the IO manager.

I was slightly off on the second point — it turns out that even MS file systems perform internal sharing checks in some cases — see the Bonus section at the end.

On top of this, using IO_IGNORE_SHARE_ACCESS_CHECK results in outright undocumented behavior when subsequently opening the same file.

Note: I am testing/disassembling on Windows 10 RS3.

Undocumented Behavior

On a couple of occasions, we’ve been surprised at the outcome of the following test:

  1. Open an existing file via FltCreateFileEx2 (or similar) with:
    • DesiredAccess = FILE_READ_DATA
  2. While the file handle is still open, open the same file with:
    • DesiredAccess = FILE_WRITE_DATA
    • ShareAccess = 0
    • Flags = 0

We expect the second open to fail with STATUS_SHARING_VIOLATION, since the file is already open for READ and the sharing flags are 0.

However, the second open succeeds. We find the same regardless of the desired access of the second open — FILE_READ_DATA and DELETE also work. We’ve reproduced this on Windows 7 SP1 as well as Windows 10 SP3.

When I finally decided to post my puzzlement to NTFSD, the site was down due to an OSR office move. So, I dove in. I later found someone else asking about this behavior but I did not come across an explanation for it in the archives.


The IO Manager provides a set of functions to make it easier for file systems to handle share access checking. In particular, IoCheckShareAccess and IoCheckShareAccessEx can be used to check share access permissions when opening a file.

Local Microsoft file systems use these functions (you can see this by examining fastfat!FatCheckShareAccess or by disassembling NTFS!NtfsCheckShareAccess). Here’s the signature for IoCheckShareAccess:

  ACCESS_MASK   DesiredAccess,
  ULONG         DesiredShareAccess,
  PFILE_OBJECT  FileObject,
  PSHARE_ACCESS ShareAccess,
  BOOLEAN       Update

MSDN explains the parameters and touches on the relevance of ShareAccess — this is a struct that is stored in the File/Directory/Stream Control Block (FCB/DCB/SCB) of a file object and is therefore common to all open instances of the file/directory/stream. Here’s the definition from a recent WDK:

typedef struct _SHARE_ACCESS {
    ULONG OpenCount;
    ULONG Readers;
    ULONG Writers;
    ULONG Deleters;
    ULONG SharedRead;
    ULONG SharedWrite;
    ULONG SharedDelete;

And, here’s a description of what each field should contain:

OpenCount The number of open file objects with (FILE_READ_DATA | FILE_EXECUTE | FILE_WRITE_DATA | FILE_APPEND_DATA | DELETE) access (for the same file/directory/stream)
Readers The number of open file objects with (FILE_READ_DATA | FILE_EXECUTE) access
Writers The number of open file objects with (FILE_WRITE_DATA | FILE_APPEND_DATA) access
Deleters The number of open file objects with DELETE access
SharedRead The number of open file objects with FILE_SHARE_READ
SharedWrite The number of open file objects with FILE_SHARE_WRITE
SharedDelete The number of open file objects with FILE_SHARE_DELETE

IoCheckShareAccess is not a huge function and so I disassembled it to see what it does (actually, I disassembled IoCheckLinkShareAccess since that’s the underlying function). IoCheckLinkAccess does the following:

  1. Set the ReadAccess, WriteAccess, and DeleteAccess bits in the FILE_OBJECT, based on the DesiredAccess parameter. If all of these are 0, return with STATUS_SUCCESS.
  2. Set the SharedRead, SharedWrite, and SharedDelete bits in the FILE_OBJECT based on the ShareAccess parameter. At this point, if the IO_IGNORE_SHARE_ACCESS_CHECK flag is present, return with STATUS_SUCCESS.
  3. Check that DesiredAccess does not collide with the share access of existing opens. If there is a collision, return STATUS_SHARING_VIOLATION.
  4. Check that DesiredSharedAccess does not collide with the access permissions of existing opens. If there is a collision, return STATUS_SHARING_VIOLATION.
  5. Finally, if the share access check passes and Update is non-zero, update the relevant counts in the common ShareAccess struct.

The key piece — if IO_IGNORE_SHARE_ACCESS_CHECK is passed in, IoCheckShareAccess skips steps 3-5. This means that the share access check will always pass for the current open, which is what we expect. Additionally, since the ShareAccess struct is not updated, share access checks for subsequent opens will not be affected by the current open!

Final Thoughts

The implementation of IO_IGNORE_SHARE_ACCESS_CHECK makes a lot of sense when you consider concurrency in the file system. After all, a call to FltCreateFileEx2 with IO_IGNORE_SHARE_ACCESS_CHECK and a a call to CreateFile can be issued at the same time. It would be horrendously frustrating if the CreateFile sometimes succeeded and sometimes failed, because of some inconsiderate kernel driver.

The documentation needs to be updated to make this behavior clear. I’ve submitted a doc issue to FltCreateFileEx2 via GitHub issues (this seems new).

Big thanks to MS for making the fastfat source available.

Bonus #1

Here’s another reason why the MSDN docs are ambiguous — even local Microsoft file systems can trigger a sharing violation when the IO_IGNORE_SHARE_ACCESS_CHECK flag is passed in.

When a file system sees IRP_MJ_CLEANUP for a file object, the counters in the associated _SHARE_ACCESS struct are decremented via IoRemoveShareAccess. However, the file object can still be used for memory-mapped IO and so the file system may perform additional sharing checks.

In particular, the fastfat FatCheckShareAccess function checks for writable sections before calling IoCheckShareAccess. Here’s the excerpt:

    //  Do an extra test for writeable user sections if the user did not allow
    //  write sharing - this is neccessary since a section may exist with no handles
    //  open to the file its based against.

    if ((NodeType( FcbOrDcb ) == FAT_NTC_FCB) &&
        !FlagOn( ShareAccess, FILE_SHARE_WRITE ) &&
        MmDoesFileHaveUserWritableReferences( &FcbOrDcb->NonPaged->SectionObjectPointers )) {


It is reasonable to assume that NTFS does the same thing, but on a cursory disassembly of NTFS!NtfsCheckShareAccess I didn’t see any calls to MmDoesFileHaveUserWriteReferences.

Bonus #2 – Where is IO_IGNORE_SHARE_ACCESS_CHECK stored?

This is just a bit of trivia really; it appears that if you need to enable this behavior without the use of the IO_IGNORE_SHARE_ACCESS_CHECK flag, a minifilter can just call IoSetFileObjectIgnoreSharing on Windows 7+. However…

At present, when the IO Manager sees the IO_IGNORE_SHARE_ACCESS_CHECK flag, it sets bit 0 in the _FILE_OBJECT->FileObjectExtension.FoExtFlags field.

   +0x04a ReadAccess       : UChar
   +0x04b WriteAccess      : UChar
   +0x04c DeleteAccess     : UChar
   +0x04d SharedRead       : UChar
   +0x04e SharedWrite      : UChar
   +0x04f SharedDelete     : UChar
   +0x0d0 FileObjectExtension : Ptr64 Void

+ 0x000 FoExtFlags : Uint4B
+ 0x008 FoExtPerTypeExtension : [9] Ptr64 Void
+ 0x050 FoIoPriorityHint : _IOP_PRIORITY_HINT

We can see this by looking at the disassembly for IoIsFileObjectIgnoringSharing, which “determines if a file object is set with the option to ignore file sharing access checks.”.

0: kd> uf IoIsFileObjectIgnoringSharing
// rax = FileObject->FileObjectExtension
fffff803`38865520 488b81d0000000  mov     rax,qword ptr [rcx+0D0h]
fffff803`38865527 4885c0          test    rax,rax
fffff803`3886552a 7505            jne     nt!IoIsFileObjectIgnoringSharing+0x11 (fffff803`38865531)  Branch

fffff803`3886552c 32c9            xor     cl,cl

fffff803`3886552e 8ac1            mov     al,cl
fffff803`38865530 c3              ret

// eax = FileObject->FileObjectExtension->FoExtFlags
fffff803`38865531 8b00            mov     eax,dword ptr [rax]
fffff803`38865533 b101            mov     cl,1
// test 1, (FileObject->FileObjectExtension->FoExtFlags & 0xf)
fffff803`38865535 84c1            test    cl,al
fffff803`38865537 74f3            je      nt!IoIsFileObjectIgnoringSharing+0xc (fffff803`3886552c)  Branch

fffff803`38865539 ebf3            jmp     nt!IoIsFileObjectIgnoringSharing+0xe (fffff803`3886552e)  Branch


Posted in File System, Reversing | Leave a comment

Finding an Exception in a user-mode minidump

I spend most of my time in kernel and so I still fumble around sometimes when asked to look at user-mode crash dumps. In particular, someone gave me an .hdmp file recently — I don’t know much about this kind of dump but it appears to be a mini-dump with some additional heap info in it.

When I opened the file, WinDBG had some bad news:

ERROR: Unable to find system thread 3DC
ERROR: The thread being debugged has either exited or cannot be accessed
ERROR: Many commands will not work properly
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
ERROR: Exception C0000005 occurred on unknown thread 3DC
(1150.3dc): Access violation - code c0000005 (first/second chance not available)

Oh well. At least .excr works.

0:???> .excr
WARNING: The debugger does not have a current process or thread
WARNING: Many commands will not work
WARNING: The debugger does not have a current process or thread
WARNING: Many commands will not work

What! WinDBG, why did you lie to me?

After messing around for a bit and accidentally fixing the problem (by running !analyze -v), I realized that WinDBG did give me a hint of what was wrong. I tried resetting the current process and things started magically working.

0:???> |s 0
WARNING: The debugger does not have a current process or thread
WARNING: Many commands will not work
00007ffc`83edff64 c3 ret
 ^ Syntax error in '|s 0'

0:000> .excr
rax=0000000000000000 rbx=000000c25ddfef48 rcx=000000c25ddfed50
rdx=000000c25ddfed40 rsi=00007ff60db99dc0 rdi=000001c6f3fd4870
rip=0000000000000000 rsp=000000c25ddfecb8 rbp=000000c25ddfed69
 r8=0000000000000001 r9=00007ff60dad3d38 r10=0000000000000000
r11=000000c25ddfec20 r12=00007ff60dae4b80 r13=0000000000000000
r14=00007ff60dad6030 r15=000001c6f3fd4870
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
00000000`00000000 ?? ???



Posted in Debugging, usermode, Windbg | Leave a comment

Different Ways to View Assembly Code

tl;dr; Compiling with /FAcs gives you useful source + assembler output.

There are lots of ways to view the compiled assembly code for a function — in this post I discuss three of them. I work exclusively on Windows so your mileage will vary on other platforms. I suppose if I had an IDA Pro license, I’d just use that.

Here is my simple program. I’m interested in seeing the assembler version of DoSomething.


void DoSomething()
  wprintf(L"Hello world!");

int wmain(int argc, wchar_t* argv[])
  return 0;

View Assembly Code Using the Debugger

Generally, if I want to disassemble something, its because I’m already in the debugger (WinDBG). For this to work, I need to have symbols available.

Here’s the debug version of DoSomething:

0:000> uf dosomething
Sandbox!DoSomething [c:\src\sandbox\sandbox.cpp @ 13]:
 13 004138f0 55 push ebp
 13 004138f1 8bec mov ebp,esp
 13 004138f3 81ecc0000000 sub esp,0C0h
 13 004138f9 53 push ebx
 13 004138fa 56 push esi
 13 004138fb 57 push edi
 13 004138fc 8dbd40ffffff lea edi,[ebp-0C0h]
 13 00413902 b930000000 mov ecx,30h
 13 00413907 b8cccccccc mov eax,0CCCCCCCCh
 13 0041390c f3ab rep stos dword ptr es:[edi]
 14 0041390e 68b0ed4100 push offset Sandbox!`string' (0041edb0)
 14 00413913 e848dbffff call Sandbox!ILT+1115(_wprintf) (00411460)
 14 00413918 83c404 add esp,4
 15 0041391b 5f pop edi
 15 0041391c 5e pop esi
 15 0041391d 5b pop ebx
 15 0041391e 81c4c0000000 add esp,0C0h
 15 00413924 3bec cmp ebp,esp
 15 00413926 e805d9ffff call Sandbox!ILT+555(__RTC_CheckEsp) (00411230)
 15 0041392b 8be5 mov esp,ebp
 15 0041392d 5d pop ebp
 15 0041392e c3 ret

Here’s what I see for the release version (DoSomething has been inlined into wmain)

0:000> uf sandbox!dosomething
Sandbox!wmain [c:\src\sandbox\sandbox.cpp @ 12]:
   12 00c11060 68ac21c100      push    offset Sandbox!`string' (00c121ac)
   13 00c11065 e8c6ffffff      call    Sandbox!wprintf (00c11030)
   13 00c1106a 83c404          add     esp,4
   14 00c1106d 33c0            xor     eax,eax
   15 00c1106f c3              ret

There are reasons why using the debugger is not always a good option:

  1. I can’t run the code in my debug environment.
  2. It can take a long time to set up a debug environment and I really just want to check something simple.
  3. The function has been inlined and is not available as an independent entity.
  4. I don’t have symbols.
  5. Depending on the compiler optimizations, the debugger can get confused.
  6. etc…

View Assembly Code Using Dumpbin

If I have object files, I can use dumpbin to extract the assembly version of the function. The command is like so:

c:\src\sandbox>dumpbin /disasm Sandbox.obj > dumpbin.txt

Note: This won’t help much for release executable because symbols have been stripped. But.. it can work on static libraries, since they are zipped archives that contain object files.

Here’s the debug output for DoSomething and is fairly similar to the WinDBG output:

?DoSomething@@YAXXZ (void __cdecl DoSomething(void)):
 00000000: 55 push ebp
 00000001: 8B EC mov ebp,esp
 00000003: 81 EC C0 00 00 00 sub esp,0C0h
 00000009: 53 push ebx
 0000000A: 56 push esi
 0000000B: 57 push edi
 0000000C: 8D BD 40 FF FF FF lea edi,[ebp-0C0h]
 00000012: B9 30 00 00 00 mov ecx,30h
 00000017: B8 CC CC CC CC mov eax,0CCCCCCCCh
 0000001C: F3 AB rep stos dword ptr es:[edi]
 0000001E: 68 00 00 00 00 push offset ??_C@_1BK@IHDEKFOI@?$AAH?$AAe?$AAl?$AAl?$AAo?$AA?5?$AAw?$AAo?$AAr?$AAl?$AAd?$AA?$CB?$AA?$AA@
 00000023: E8 00 00 00 00 call _wprintf
 00000028: 83 C4 04 add esp,4
 0000002B: 5F pop edi
 0000002C: 5E pop esi
 0000002D: 5B pop ebx
 0000002E: 81 C4 C0 00 00 00 add esp,0C0h
 00000034: 3B EC cmp ebp,esp
 00000036: E8 00 00 00 00 call __RTC_CheckEsp
 0000003B: 8B E5 mov esp,ebp
 0000003D: 5D pop ebp
 0000003E: C3 ret

Here’s the release version:

Dump of file Sandbox.obj


Uh-oh. Dumpbin can’t help me as I’ve turned on whole program optimization.  The docs have this to say:

Only the /HEADERS DUMPBIN option is available for use on files produced by the /GL (Whole program optimization) compiler option.+

Back to WinDBG output? Nope, one more option…

View Assembly Code Using /FAcs Compiler Flags

If I were thinking ahead, or if I am able to rebuild from source, my colleague pointed out that you can generate very nice source + assembly at compile time using the /FAcs compiler flag.

This option will result in an extra file with a .cod suffix. The file contains machine code, assembly code, and source code! Clearly, this is the most verbose and quickest to decipher option.

In Visual Studio 2017, this option is under Properites -> C/C++ -> Output Files -> Assembler Output.


Here’s the debug version:

; Function compile flags: /Odtp /RTCsu /ZI
; File c:\src\sandbox\sandbox.cpp
; COMDAT ?DoSomething@@YAXXZ
?DoSomething@@YAXXZ PROC ; DoSomething, COMDAT

; 13 : {

00000 55 push ebp
 00001 8b ec mov ebp, esp
 00003 81 ec c0 00 00
 00 sub esp, 192 ; 000000c0H
 00009 53 push ebx
 0000a 56 push esi
 0000b 57 push edi
 0000c 8d bd 40 ff ff
 ff lea edi, DWORD PTR [ebp-192]
 00012 b9 30 00 00 00 mov ecx, 48 ; 00000030H
 00017 b8 cc cc cc cc mov eax, -858993460 ; ccccccccH
 0001c f3 ab rep stosd

; 14 : wprintf(L"Hello world!");

0001e 68 00 00 00 00 push OFFSET ??_C@_1BK@IHDEKFOI@?$AAH?$AAe?$AAl?$AAl?$AAo?$AA?5?$AAw?$AAo?$AAr?$AAl?$AAd?$AA?$CB?$AA?$AA@
 00023 e8 00 00 00 00 call _wprintf
 00028 83 c4 04 add esp, 4

; 15 : }

0002b 5f pop edi
 0002c 5e pop esi
 0002d 5b pop ebx
 0002e 81 c4 c0 00 00
 00 add esp, 192 ; 000000c0H
 00034 3b ec cmp ebp, esp
 00036 e8 00 00 00 00 call __RTC_CheckEsp
 0003b 8b e5 mov esp, ebp
 0003d 5d pop ebp
 0003e c3 ret 0
?DoSomething@@YAXXZ ENDP ; DoSomething

Here is the release version:

; Function compile flags: /Ogtp
 ; File c:\src\sandbox\sandbox.cpp
 ; COMDAT ?DoSomething@@YAXXZ
 ?DoSomething@@YAXXZ PROC ; DoSomething, COMDAT

; 8 : wprintf(L"Hello world!");

00000 68 00 00 00 00 push OFFSET ??_C@_1BK@IHDEKFOI@?$AAH?$AAe?$AAl?$AAl?$AAo?$AA?5?$AAw?$AAo?$AAr?$AAl?$AAd?$AA?$CB?$AA?$AA@
 00005 e8 00 00 00 00 call _wprintf
 0000a 59 pop ecx

; 9 : }

0000b c3 ret 0
 ?DoSomething@@YAXXZ ENDP ; DoSomething
Posted in Debugging | Leave a comment

Preventing System DLL Sideload Attacks

Problem Statement

DLL sideloading is an attack where an unintended DLL is loaded, resulting in unintended code execution. This attack is possible for any DLL; in this article, we focus on Windows system DLLs.

Sideloading system DLLs

There’s an inherent problem when loading a Windows system DLL like winhttp.dll. For compatibility reasons, the default Windows DLL search begins in the directory where the executable was launched. An attacker need only copy a rogue version of the system DLL to the same directory as the executable. That’s pretty easy even without elevated privileges — an attacker can copy the executable + the rogue DLL to the temp folder, and execute the binary from there.

This vulnerability is very common when loading Windows system DLLs because we usually omit the full path for these modules.

  • Compile-time dynamically-linked libraries are resolved by DLL name alone before execution begins. This is not something we can change.
  • When loading system DLLs dynamically at runtime, we generally pass just the DLL name and let the loader figure out where the system directory is.
  • We generally skip trust checks on system DLLs since a) we don’t have the full path, and b) we rely on system functions like WinVerifyTrust to validate the signature. If the operating system has been compromised already, the trust check won’t really give us much confidence.

Sideloading is especially effective for executables that run with elevated privileges and are embedded signed. For example, many users will quickly agree to elevate privileges if the UAC prompts tells them that their trusted anti-virus vendor wants to make changes.

This is less of an issue for Microsoft executables since they are usually catalog signed. This means that moving them to another location breaks trust. For example, here’s the dialog when we copy mmc.exe out of the system directory and execute it:

This is not a new attack vector, but it’s worth understanding the nuances and mitigations.

MSDN has a good amount of information about this topic but it is confusing and spread out over multiple articles. When not otherwise specified, the details in this article come from the MSDN article entitled Dynamic-Link Library Search Order. Additional details are provided where appropriate.

Searching for DLLs

By default, when a DLL load is done by name alone, the default DLL search path is:

  • The directory from which the application loaded.
  • The system directory.
  • The 16-bit system directory.
  • The Windows directory.
  • The current directory.
  • The directories that are listed in the PATH environment variable.

There are a couple of exceptions:

  1. If a DLL with the same name is already loaded in the process, the existing DLL is used.
  2. If the DLL is in the list of Known DLLs, it has already been loaded at boot time and so the existing DLL is used.The list of known DLLs is taken from HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\KnownDLLs and is then augmented by additional DLLs that are statically linked to the DLLs in the list. For more information, see Larry Osterman’s article entitled What are Known DLLs anyway?.

    If you use WinObj to view the Object directory, you can see the full list of Known DLLs on a system (and yes, the list of Known DLLs is an attack vector but it is out of scope for this article). Note that this list varies by operating system. For example, on Windows Vista, crypt32.dll is not in the Known DLLs list.


Note: The mitigations presented in this section are only applicable on Windows Vista SP1+ and require the critical Windows security update specified in KB2533623 on pre-Win8 machines. This security update was released in 2011 and it is quite reasonable to expect a machine patched to this level.

Explicit System DLL Loads

In all places where your code explicitly loads a system DLL via LoadLibrary or LoadLibraryEx, ensure that you use LoadLibraryEx and pass the LOAD_LIBRARY_SEARCH_SYSTEM32 flag. This ensures that only the system directory is searched for that specific DLL.

Note: In our testing, we found that passing the LOAD_LIBRARY_SEARCH_SYSTEM32 flag on unpatched Vista / Win7 systems can cause LoadLibraryEx to fail. If you need to support pre-KB2533623 versions of Vista/Win7, you will have to do a little more work. One good option is to look for the existence of the SetDefaultDllDirectories function, which was released in the same KB. If the function exists, it is safe to pass the LOAD_LIBRARY_SEARCH_SYSTEM32 flag.

Implicit DLL Loads

Calls to functions like WinVerifyTrust end up loading additional system DLLs and you have no way to control the LoadLibraryEx flags in this case.

To prevent this you can call the SetDefaultDllDirectories function at the start of your executable and pass in the LOAD_LIBRARY_SEARCH_SYSTEM32 flag. After this, the loader will only search the system directory unless lpFileName specifies a full path and/or override flags are passed into LoadLibraryEx.

After making this change, any loads of non-system DLLs must have the full path or non-zero LoadLibraryEx flags. Passing in a full path is a good practice in any case, as is doing a trust check on the DLL before loading it.

Note: Since SetDefaultDllDirectories is not available on unpatched Vista/Win7 systems it is a really good idea to dynamically load it from kernel32.dll.

Compile-Time DLL Imports

Some DLLs linked at compile-time (like kernel32.dll) will be loaded from the Known DLLs list.

However, there are many DLLs that are not on the Known DLLs list. This includes DLLs like winhttp.dll, sfc.dll, version.dll, lz32.dll and more.

In this case, you have two options:

  1. Switch to manually loading these DLLs and use GetProcAddress to get the imports.
  2. Mark the library for delay loading at link-time. This defers the DLL load to the first time one of its exports is called. Before the first call into the DLL, ensure that your executable has already called SetDefaultDllDirectories with the LOAD_LIBRARY_SEARCH_SYSTEM32 flag.The linker option for this is /DELAYLOAD:<dllname>.dll. You can find various pages on MSDN with this information, including this one.

    Note: You still link with the <dllname>.lib file and you also need to link with delayimp.lib as it handles the work of loading the DLL at runtime. Ex: for winhttp.dll, the linker options would be:
    winhttp.lib delayload.imp /DELAYLOAD:winhttp.dll


If someone forces you at gunpoint to support pre-Win8 systems without KB2533623 installed, you can get most of the way there. Keep in mind that there is no way to handle the case where a function like WinVerifyTrust dynamically loads additional DLLs as part of function execution.

  1. Always pass in the full path to LoadLibraryEx. For system DLLs, you can get the system directory using the GetSystemDirectory function.
  2. Always pass in the LOAD_WITH_ALTERED_SEARCH_PATH flag to LoadLibraryEx. This flag was introduced in Windows XP and helps with the following case:
    1. Dynamically load c:\windows\system32\foo.dll using LoadLibraryEx.
    2. Foo.dll is linked with bar.dll at compile time, and bar is also a system DLL.
      Without the LOAD_WITH_ALTERED_SEARCH_PATH, the search for bar.dll still begins in the executable directory. With the LOAD_WITH_ALTERED_SEARCH_PATH flag, the search path begins with the path to foo.dll, i.e. the system directory.
  3. Load DLLs manually at run-time wherever possible, and use GetProcAddress to get pointers to the required functions. DLLs like kernel32.dll and ntdll.dll will still be loaded when the executable starts, but they are on the Known DLLs list so that is not a problem.
Posted in Security | Tagged , | Leave a comment

MEX is Your Friend: Analyzing 32-bit Processes in a 64-bit Kernel Dump


In 2016, Microsoft publicly released a WinDBG extension called MEX. It has a number of commands that make life easier when kernel debugging or analyzing a crash dumps. You can download it here if you want to give it a try.

The Problem

I was recently asked to look a hang issue in Microsoft Outlook. I was given a full memory dump taken on a 64-bit system, which seemed like overkill (it wasn’t). It’s generally quite straightforward to analyze user mode apps from a memory dump — see the following OSR article for more info: Analyst’s Perspective: Analyzing User Mode State from a Kernel Connection.

I started my debugging session and quickly ran into a snag: Outlook was running as 32-bit process, leading to stacks with mostly wow64* calls in them.

0: kd> !process 0 0 outlook.exe
PROCESS fffffa800b6e6060
 SessionId: 1 Cid: 1408 Peb: 7efdf000 ParentCid: 1268
 DirBase: a9baa000 ObjectTable: fffff8a0083de430 HandleCount: 6880.

0: kd> .process /r /p fffffa800b6e6060
Implicit process is now fffffa80`0b6e6060
Loading User Symbols
0: kd> !process fffffa800b6e6060
PROCESS fffffa800b6e6060
 SessionId: 1 Cid: 1408 Peb: 7efdf000 ParentCid: 1268
 DirBase: a9baa000 ObjectTable: fffff8a0083de430 HandleCount: 6880.
 THREAD fffffa800b858060 Cid 1408.180c Teb: 000000007efaa000 Win32Thread: fffff900c1d3f010 WAIT: (UserRequest) UserMode Non-Alertable
 fffff880`093dbec0 fffff800`03484e42 nt!KiSwapContext+0x7a
 fffff880`093dc000 fffff800`034918da nt!KiCommitThreadWait+0x1d2
 fffff880`093dc090 fffff800`0378b1af nt!KeWaitForMultipleObjects+0x272
 fffff880`093dc350 fffff800`037b8fc9 nt!ObpWaitForMultipleObjects+0x294
 fffff880`093dc820 fffff800`0348e693 nt!NtWaitForMultipleObjects32+0xec
 fffff880`093dca70 00000000`74cc2e09 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`093dcae0)
 00000000`0352ebe8 00000000`74cc283e wow64cpu!CpupSyscallStub+0x9
 00000000`0352ebf0 00000000`74d3d286 wow64cpu!WaitForMultipleObjects32+0x3b
 00000000`0352ecb0 00000000`74d3c69e wow64!RunCpuSimulation+0xa
 00000000`0352ed00 00000000`773898ec wow64!Wow64LdrpInitialize+0x42a
 00000000`0352f250 00000000`7734a36e ntdll! ?? ::FNODOBFM::`string'+0x22b74
 00000000`0352f2c0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

The Painful Solution

After asking around, I found out that I could use .thread /w to look at 32-bit stacks but the method proved cumbersome, as I was looking for a particular thread out of 57 threads in the process. I was too lazy to write a script and so I spent over an hour going through each stack in the process like so:

0: kd> .thread /w fffffa800b858060 ; kb ; .effmach amd64
Implicit thread is now fffffa80`0b858060
x86 context set
 *** Stack trace for last set context - .thread/.cxr resets it
 # ChildEBP RetAddr Args to Child 
00 039afa64 755e171a 00000005 039afab4 00000001 ntdll_77500000!NtWaitForMultipleObjects+0x15
01 039afb00 76c019fc 039afab4 039afb28 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100
02 039afb48 76a90882 00000005 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0
03 039afb9c 6b16df8b 000003a0 039afbe4 ffffffff USER32!RealMsgWaitForMultipleObjectsEx+0x14d
WARNING: Stack unwind information not available. Following frames may be wrong.
04 039afbbc 2f7a5538 00000004 039afbe4 00000000 mso!MsoHrSetupHTMLImport+0x392
05 039afc10 62398646 00000000 0775e2d4 623985a6 OUTLOOK!HrMsgDownloadedNotification+0x3ff
06 039afc34 6b0a08e6 0775e2d4 00000000 572383a4 olmapi32!MSProviderInit+0x95f
07 039afc78 6b09e0fd 1191f900 039afd10 0039d16c mso!Ordinal3464+0x4a2
08 039afc98 6b09ddc3 039afd10 00000000 039afcf4 mso!Ordinal2771+0x450
09 039afcb4 6b09c027 039afd10 00000000 003997f0 mso!Ordinal2771+0x116
0a 039afce8 6b0962b7 003997f0 00000000 6b0962b7 mso!Ordinal2929+0x209
0b 039afd44 76c0336a 003997f0 039afd90 77539902 mso!Ordinal4724+0x67
0c 039afd50 77539902 003997f0 74fcff46 00000000 kernel32!BaseThreadInitThunk+0xe
0d 039afd90 775398d5 6b09625f 003997f0 ffffffff ntdll_77500000!__RtlUserThreadStart+0x70
0e 039afda8 00000000 6b09625f 003997f0 00000000 ntdll_77500000!_RtlUserThreadStart+0x1b
Effective machine: x64 (AMD64)

The MEX Solution

My solution worked but it was incredibly inefficient. The next morning I was still looking at the dump and decided to play around with MEX to see if it could help me out with something else. As it turned out, it was also able to help with my initial problem. Instead of spending an hour+ pasting commands into the debugger, I was able to get to the same point in about 1 minute.

0: kd> !mex.p fffffa800b6e6060
Name Address Ses PID Parent PEB Create Time Mods Handle Thrd User Name
=========== ======================== === ============= ============= ================ ========================== ==== ====== ==== ==========
OUTLOOK.EXE fffffa800b6e6060 (E|K|O) 1 1408 (0n5128) 1268 (0n4712) 000000007efdf000 04/26/2017 03:52:59.367 AM 281 6880 57 BOS\zy10os

Command Line: "C:\Program Files (x86)\Microsoft Office\Office14\OUTLOOK.EXE"

Memory Details:

VM Peak Work Set Commit Size PP Quota NPP Quota
 ========= ========= ========= =========== ======== =========
 963.62 MB 980.07 MB 281.02 MB 221.82 MB 1.36 MB 206 KB

Show LPC Port information for process

Show Threads: Unique Stacks !mex.listthreads (!lt) fffffa800b6e6060 !process fffffa800b6e6060 7

0: kd> ! fffffa800b6e6060
Process PID Thread Id State Time Reason Waiting On
=============== ==== ================ ==== ======= ============= ============== =================================================
OUTLOOK.EXE *32 1408 fffffa8007592060 fc0 Waiting 11s.138 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8007588840 1814 Waiting 9s.001 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b4dbb50 14bc Waiting 15ms UserRequest 
OUTLOOK.EXE *32 1408 fffffa80082c3b50 1808 Waiting 15ms UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b858060 180c Waiting 11s.138 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b76bb50 11bc Waiting 15ms WrQueue 
OUTLOOK.EXE *32 1408 fffffa800ad06b50 1910 Waiting 55s.551 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b552b20 1544 Waiting 11s.122 WrUserRequest 
OUTLOOK.EXE *32 1408 fffffa8007511060 15a4 Waiting 11s.122 WrUserRequest 
OUTLOOK.EXE *32 1408 fffffa8007404060 1670 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b71ab50 1930 Waiting 36m:05.449 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8007578060 1a5c Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8007581b50 1a0c Waiting 1m:28.733 WrQueue 
OUTLOOK.EXE *32 1408 fffffa8008fe9060 11b8 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b85b060 1590 Waiting 1s.747 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800a1a4060 cbc Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa80072f3b50 1820 Waiting 9s.984 WrLpcReply Thread: fffffa800c877060 in mcshield.exe (0n3120)
OUTLOOK.EXE *32 1408 fffffa80074a4b50 1228 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8007584b50 58c Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8009af1060 15c4 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800983e7c0 648 Waiting 6m:18.598 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b3b24b0 16b4 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b87f060 16d8 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800adc5060 1668 Waiting 1s.778 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b67e6c0 1788 Waiting 1m:39.949 UserRequest 
OUTLOOK.EXE *32 1408 fffffa80075a5b50 1658 Waiting 38s.017 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b52eb50 c20 Waiting 21m:28.505 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b40cb50 bd4 Waiting 15ms UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b375060 e00 Waiting 1m:31.993 WrUserRequest 
OUTLOOK.EXE *32 1408 fffffa800b2d67f0 10c0 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b6d8060 1990 Waiting 3m:59.196 WrUserRequest 
OUTLOOK.EXE *32 1408 fffffa800ba53b50 16e0 Waiting 6m:18.598 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800baa1060 1a98 Waiting 9m:41.197 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b697060 14a8 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800ba2e060 15ec Waiting 1m:30.511 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800bb68060 1924 Waiting 1m:30.511 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800ba74060 1130 Waiting 35s.271 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800bad2060 3e0 Waiting 1m:31.744 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b700060 1900 Waiting 11s.122 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8007362640 18a0 Waiting 1m:29.528 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800bab5060 ce0 Waiting 1m:30.932 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8009e3f060 15fc Waiting 15h:22:24.615 WrQueue 
OUTLOOK.EXE *32 1408 fffffa800be1f640 1db8 Waiting 1m:31.416 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800689f060 1784 Waiting 6m:51.905 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8006b59710 1f88 Waiting 44s.304 WrUserRequest 
OUTLOOK.EXE *32 1408 fffffa800bd60900 21a0 Waiting 15m:41.044 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8006c70060 1c5c Waiting 15ms UserRequest 
OUTLOOK.EXE *32 1408 fffffa800c253710 1850 Waiting 280ms UserRequest 
OUTLOOK.EXE *32 1408 fffffa800befbb50 2074 Waiting 55s.661 DelayExecution 
OUTLOOK.EXE *32 1408 fffffa8009e95840 2164 Waiting 9s.001 WrQueue 
OUTLOOK.EXE *32 1408 fffffa800c71a650 22b8 Waiting 4m:01.567 WrQueue 
OUTLOOK.EXE *32 1408 fffffa800c000870 2314 Waiting 15ms WrQueue 
OUTLOOK.EXE *32 1408 fffffa8009ede2b0 2014 Waiting 15ms WrQueue 
OUTLOOK.EXE *32 1408 fffffa800bef7460 1d24 Waiting 15ms WrQueue 
OUTLOOK.EXE *32 1408 fffffa800c28f060 1848 Waiting 11s.232 WrQueue 
OUTLOOK.EXE *32 1408 fffffa800c0a2b50 1258 Waiting 11s.122 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800bfde060 21a8 Waiting 15ms WrQueue

Thread Count: 57

0: kd> !mex.t fffffa800b858060
Process Thread CID TEB UserTime KernelTime ContextSwitches Wait Reason Time State COM-Initialized
OUTLOOK.EXE *32 (fffffa800b6e6060) fffffa800b858060 (E|K|W|R|V) 1408.180c 000000007efaa000 312ms 94ms 73230 UserRequest 11s.138 Waiting APTKIND_MULTITHREADED (MTA)


# Child-SP Return Call Site
0 fffff880093dbec0 fffff80003484e42 nt!KiSwapContext+0x7a
1 fffff880093dc000 fffff800034918da nt!KiCommitThreadWait+0x1d2
2 fffff880093dc090 fffff8000378b1af nt!KeWaitForMultipleObjects+0x272
3 fffff880093dc350 fffff800037b8fc9 nt!ObpWaitForMultipleObjects+0x294
4 fffff880093dc820 fffff8000348e693 nt!NtWaitForMultipleObjects32+0xec
5 fffff880093dca70 0000000074cc2e09 nt!KiSystemServiceCopyEnd+0x13
0 00000000039afa64 00000000755e171a ntdll_77500000!NtWaitForMultipleObjects+0x15
1 00000000039afa6c 0000000076c019fc KERNELBASE!WaitForMultipleObjectsEx+0x100
2 00000000039afb08 0000000076a90882 kernel32!WaitForMultipleObjectsExImplementation+0xe0
3 00000000039afb50 000000006b16df8b USER32!RealMsgWaitForMultipleObjectsEx+0x14d
4 00000000039afba4 000000002f7a5538 mso!MsoHrSetupHTMLImport+0x392
5 00000000039afbc4 0000000062398646 OUTLOOK!HrMsgDownloadedNotification+0x3ff
6 00000000039afc18 000000006b0a08e6 olmapi32!MSProviderInit+0x95f
7 00000000039afc3c 000000006b09e0fd mso!Ordinal3464+0x4a2
8 00000000039afc80 000000006b09ddc3 mso!Ordinal2771+0x450
9 00000000039afca0 000000006b09c027 mso!Ordinal2771+0x116
a 00000000039afcbc 000000006b0962b7 mso!Ordinal2929+0x209
b 00000000039afcf0 0000000076c0336a mso!Ordinal4724+0x67
c 00000000039afd4c 0000000077539902 kernel32!BaseThreadInitThunk+0xe
d 00000000039afd58 00000000775398d5 ntdll_77500000!__RtlUserThreadStart+0x70
e 00000000039afd98 0000000000000000 ntdll_77500000!_RtlUserThreadStart+0x1b

Switch to x86 - to switch back to x64/amd64 run !sw


MEX is your friend.

Posted in Debugging, Windbg | Leave a comment

Windows Subystem For Linux – File Reads

I’ve read a bit about the way that Microsoft has implemented the Windows Subsystem for Linux (WSL). A one sentence summary: all system calls in a WSL process are fulfilled by a special kernel driver called a pico provider. I wanted to flesh out this knowledge from the perspective of a minifilter sitting on the file system stack.

For this research, I am using the following Windows 10 RS2 preview build.
Windows 10 Kernel Version 15007 MP (2 procs) Free x64
Built by: 15007.1000.amd64fre.rs_prerelease.170107-1846

Disclaimer: This pre-release of Windows 10 RS2 was a publicly available build and I only used publicly available information in putting together this article.


Here is an article on how to get bash up and running on Windows 10 RS1 and later. One major point that I initially missed is that this is only supported on 64-bit versions of Windows.

Mostly Normal

Once everything is up and running, the bash process appears in Task Manager as normal.


First, let’s use Procmon to verify that WSL does use the normal file system stack.

$ read < "a.txt"


That’s a good start but I had two questions after seeing the above output:

  1. Why is the process name missing?
  2. Why is there a CREATE with no permissions requested?

Pico Processes

I’m using v3.31 of Procmon and it’s not able to show the process name for a WSL process. We know that WSL processes are implemented as pico processes so this makes some sense. Even WinDBG gets confused:

0: kd> !process 0n440
Searching for Process with Cid == 1b8
PROCESS ffffa90bbef49080
 SessionId: 1 Cid: 01b8 Peb: 00000000 ParentCid: 074c
 DirBase: 79ab9000 ObjectTable: ffffc00afb7b7540 HandleCount: 0.
 Image: System Process
 VadRoot ffffa90bc07b7960 Vads 370 Clone ffffa90bc0d530f0 Private 88. Modified 924. Locked 1.
 DeviceMap 0000000000000000
 Token ffffc00afa9f0960
 ElapsedTime 00:39:46.612
 UserTime 00:00:00.000
 KernelTime 00:00:00.000
 THREAD ffffa90bbd889080 Cid 01b8.1b24 Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
 ffffa90bbeef81e0 NotificationEvent
 ffffa90bc0409bd0 NotificationEvent
 Not impersonating
 Owning Process ffffa90bbef49080 Image: System Process
 Attached Process N/A Image: N/A
 Wait Start TickCount 299722 Ticks: 2439 (0:00:00:38.109)
 Context Switch Count 1553 IdealProcessor: 1 
 UserTime 00:00:00.031
 KernelTime 00:00:00.906
 Stack Init ffffd400f8378c90 Current ffffd400f8378310
 Base ffffd400f8379000 Limit ffffd400f8373000 Call 0000000000000000
 Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
 Child-SP RetAddr Call Site
 ffffd400`f8378350 fffff801`7fa77867 nt!KiSwapContext+0x76
 ffffd400`f8378490 fffff801`7fa76f90 nt!KiSwapThread+0x477
 ffffd400`f8378540 fffff801`7fa76583 nt!KiCommitThreadWait+0x160
 ffffd400`f83785e0 fffff80b`b452e1b6 nt!KeWaitForMultipleObjects+0x203
 ffffd400`f83786c0 fffff80b`b452e379 LXCORE!LxpThreadWait+0x126
 ffffd400`f8378770 fffff80b`b44c6d38 LXCORE!LxpThreadWaitEx+0x49
 ffffd400`f83787d0 fffff80b`b45480e4 LXCORE!LxpDevTerminalRead+0x170
 ffffd400`f8378860 fffff80b`b45251e1 LXCORE!VfsFileRead+0x1b4
 ffffd400`f8378900 fffff80b`b451e22c LXCORE!LxpSyscall_READ+0x101
 ffffd400`f83789d0 fffff80b`b453be76 LXCORE!LxpSysDispatch+0x16c
 ffffd400`f8378aa0 fffff801`800ac2c0 LXCORE!PicoSystemCallDispatch+0x16
 ffffd400`f8378ad0 fffff801`7fb8a392 nt!PsPicoSystemCallDispatch+0x20
 ffffd400`f8378b00 00007f06`a24e69b0 nt!KiSystemServiceUser+0x82 (TrapFrame @ ffffd400`f8378b00)
 00007fff`fcb27498 00000000`004aee97 0x00007f06`a24e69b0
 00007fff`fcb274a0 00000000`00000000 0x4aee97

Just for confirmation, this is indeed a Pico Process and so LXCORE must be the Pico Provider.

0: kd> dt _EPROCESS ffffa90bbf26a500

 +0x6d4 Minimal : 0y1
 +0x718 PicoContext : 0xffffc00a`fa03c000 Void
 +0x818 PicoCreated : 0y1

Bash Process

There is actually a win32 bash.exe process on the system but my understanding is that it is only responsible for launching the WSL pico processes.

PROCESS ffffa90bc17b9340
 SessionId: 1 Cid: 0c4c Peb: 4ac55f3000 ParentCid: 0f80
 DirBase: 1ebf2000 ObjectTable: ffffc00afbb466c0 HandleCount: 90.
 Image: bash.exe

0: kd> dt _EPROCESS ffffa90bc17b9340
 +0x6d4 Minimal : 0y0

I/O Pattern

In the Procmon output, we see two CREATE operations. The first CREATE opens the file without any permissions and the second opens the same file with a number of permissions. As it turns out, the two CREATE operations are very much related and not in an obvious way.

I used WinDBG to set a conditional breakpoint in my minifilter PreCreate operation:

bp Myfilter!PreCreate ".if (ffffa90bbef49080== $proc) {} .else {gc}"

The file object in the first CREATE operation looks completely normal. This is a relative open to a directory that has already been opened (my Procmon filter excluded this original open).

1: kd> dx -r1 (*((FLTMGR!_FILE_OBJECT *)0xffffa90bc183b530))
(*((FLTMGR!_FILE_OBJECT *)0xffffa90bc183b530)) [Type: _FILE_OBJECT]
 [+0x000] Type : 5 [Type: short]
 [+0x002] Size : 216 [Type: short]
 [+0x008] DeviceObject : 0xffffa90bbe7c8b50 : Device for "\Driver\volmgr" [Type: _DEVICE_OBJECT *]
 [+0x010] Vpb : 0x0 [Type: _VPB *]
 [+0x018] FsContext : 0x0 [Type: void *]
 [+0x020] FsContext2 : 0x0 [Type: void *]
 [+0x028] SectionObjectPointer : 0x0 [Type: _SECTION_OBJECT_POINTERS *]
 [+0x030] PrivateCacheMap : 0x0 [Type: void *]
 [+0x038] FinalStatus : 0 [Type: long]
 [+0x040] RelatedFileObject : 0xffffa90bc19e3df0 [Type: _FILE_OBJECT *]
 [+0x048] LockOperation : 0x0 [Type: unsigned char]
 [+0x049] DeletePending : 0x0 [Type: unsigned char]
 [+0x04a] ReadAccess : 0x0 [Type: unsigned char]
 [+0x04b] WriteAccess : 0x0 [Type: unsigned char]
 [+0x04c] DeleteAccess : 0x0 [Type: unsigned char]
 [+0x04d] SharedRead : 0x0 [Type: unsigned char]
 [+0x04e] SharedWrite : 0x0 [Type: unsigned char]
 [+0x04f] SharedDelete : 0x0 [Type: unsigned char]
 [+0x050] Flags : 0x20002 [Type: unsigned long]
 [+0x058] FileName : "a.txt" [Type: _UNICODE_STRING]
 [+0x068] CurrentByteOffset : {0} [Type: _LARGE_INTEGER]
 [+0x070] Waiters : 0x0 [Type: unsigned long]
 [+0x074] Busy : 0x0 [Type: unsigned long]
 [+0x078] LastLock : 0x0 [Type: void *]
 [+0x080] Lock [Type: _KEVENT]
 [+0x098] Event [Type: _KEVENT]
 [+0x0b0] CompletionContext : 0x0 [Type: _IO_COMPLETION_CONTEXT *]
 [+0x0b8] IrpListLock : 0x0 [Type: unsigned __int64]
 [+0x0c0] IrpList [Type: _LIST_ENTRY]
 [+0x0d0] FileObjectExtension : 0x0 [Type: void *]

The file object in the second CREATE is also a relative open, but relative to the file itself!

0: kd> dx -r1 (*((FLTMGR!_FILE_OBJECT *)0xffffa90bbe60bef0))
(*((FLTMGR!_FILE_OBJECT *)0xffffa90bbe60bef0)) [Type: _FILE_OBJECT]
 [+0x000] Type : 5 [Type: short]
 [+0x002] Size : 216 [Type: short]
 [+0x008] DeviceObject : 0xffffa90bbe7c8b50 : Device for "\Driver\volmgr" [Type: _DEVICE_OBJECT *]
 [+0x010] Vpb : 0x0 [Type: _VPB *]
 [+0x018] FsContext : 0x0 [Type: void *]
 [+0x020] FsContext2 : 0x0 [Type: void *]
 [+0x028] SectionObjectPointer : 0x0 [Type: _SECTION_OBJECT_POINTERS *]
 [+0x030] PrivateCacheMap : 0x0 [Type: void *]
 [+0x038] FinalStatus : 0 [Type: long]
 [+0x040] RelatedFileObject : 0xffffa90bc183b530 [Type: _FILE_OBJECT *] <-- Hey look, it's the first file object!
 [+0x048] LockOperation : 0x0 [Type: unsigned char]
 [+0x049] DeletePending : 0x0 [Type: unsigned char]
 [+0x04a] ReadAccess : 0x0 [Type: unsigned char]
 [+0x04b] WriteAccess : 0x0 [Type: unsigned char]
 [+0x04c] DeleteAccess : 0x0 [Type: unsigned char]
 [+0x04d] SharedRead : 0x0 [Type: unsigned char]
 [+0x04e] SharedWrite : 0x0 [Type: unsigned char]
 [+0x04f] SharedDelete : 0x0 [Type: unsigned char]
 [+0x050] Flags : 0x20002 [Type: unsigned long]
 [+0x058] FileName : "" [Type: _UNICODE_STRING]
 [+0x068] CurrentByteOffset : {0} [Type: _LARGE_INTEGER]
 [+0x070] Waiters : 0x0 [Type: unsigned long]
 [+0x074] Busy : 0x0 [Type: unsigned long]
 [+0x078] LastLock : 0x0 [Type: void *]
 [+0x080] Lock [Type: _KEVENT]
 [+0x098] Event [Type: _KEVENT]
 [+0x0b0] CompletionContext : 0x0 [Type: _IO_COMPLETION_CONTEXT *]
 [+0x0b8] IrpListLock : 0x0 [Type: unsigned __int64]
 [+0x0c0] IrpList [Type: _LIST_ENTRY]
 [+0x0d0] FileObjectExtension : 0x0 [Type: void *]

It is perfectly legal to open a file relative to itself, but it’s also easy to mistake this pattern for a volume open. Alex Carp has a great article that helped me understand the pattern a little better. An excerpt:

reopen (FileObject->FileName is empty (Length == 0 and Buffer == NULL) but RelatedFileObject is not null). This is used when the caller wants to open a new handle to an existing FILE_OBJECT. This is not the same as opening a new handle to the existing FILE_OBJECT (duplicating the handle) because the end result of this is to open a new FILE_OBJECT for the same underlying stream, and the two FILE_OBJECTs are not linked in any other way (for example, a filter might want to open its own handle to a user file without bothering to figure out if the user has enough access (if the minifilter might want to write to the file and the original handle didn’t allow it then duplicating the handle is more complicated) or without interfering with the additional information stored in the FILE_OBJECT (like the current pointer position) and so on. According to the FASTFAT source code this should work for volume opens as well. This actually happens occasionally so filters should be prepared to deal with it.


I’m happy that things look fairly normal. The WSL read pattern is a little strange, and may require some tweaking in a minifilter that tries to detect volume opens.

Posted in Debugging | Leave a comment

Why does Notepad++ hang when I open it?

I am generally happy with Notepad++ but every now and then I find that it takes a long time to open. Yesterday, I started seeing a 20-second hang and after opening Notepad++ a few times, I became frustrated enough to troubleshoot the problem. The good news is that it took about 10 minutes to figure out the problem, which is a good ROI — once I open Notepad++ about 30 more times, I will have made back the time spent fixing it.

You can read the basics of using Windows Performance Recorder (WPR) and Windows Performance Analyzer (WPA) for wait analysis here: Why does Explorer hang when I right-click on a folder? If you’ve never done this before, it may take a couple of hours to get everything set up but it’s a one-time task.

Let’s pick up after I’ve already collected the trace and opened it in WPA. My first step is to look at the System Activity\UI Delays graph, which shows that Notepad++ is failing to check for Window messages for 21.9 seconds. That matches the delay I’m seeing in opening it.


Next, I load up the Computation\CPU Usage (Precise) graph, since it has information on every context switch.  From the UI Delays graph, I know that the hanging UI thread ID is 94092 so I set up my table as below.

notepad++_hang_cpu_precise_startIn the table, we see that the total wait time for the thread is about what we expect – 21 seconds. The longest wait is about 1 second. Next, I drill down into the thread, each time following the stack with the longest total wait time, eventually stopping when I find something “interesting”:


Above, we see XMLTools-borneo3.dll is calling the WinSock select function. MSDN says that “The select function determines the status of one or more sockets, waiting if necessary, to perform synchronous I/O.” It does appear that the function is waiting, since we see the call to WaitForSingleObject.

The other strange thing is that borneo3 is the name of my home machine. A quick look at the System Activity\Images table shows me that XMLTools-borneo3.dll is being loaded from the Notepad++ plugins directory.

At this point, I connect the dots. I use OneDrive to sync my tools across different machines. Evidently, it had a conflict when syncing and backed up the XMLTools.dll to a new name.

Once I deleted the XMLTools-borneo3.dll, the hang during open went away. Of course, it’s not clear exactly what the problem is or why it doesn’t happen with the original XMLTools.dll but as the gambler says, you gotta know when to fold them. Back to work!

Posted in Hang, Performance, Troubleshooting | Leave a comment