Oplock Cheat Sheet

I am happy to say that I deal with oplock issues very rarely. The one problem with this is that I forget the subtleties of oplocks without fail. This is meant to be a cheat sheet for myself, in a public place so that others might possibly benefit.

The Basics – What no STATUS_SHARING_VIOLATION?

Since this is public, it would be rude to continue without a basic idea of what oplocks are.

Originally, oplocks were designed as way to increase performance over the network. For example, if you know that nobody is modifying a file on a network share, you can cache your a local copy of the file for as long as you want. A lot of the Microsoft docs still talk about oplocks in this way.

In “recent” years (since Windows 7), oplocks are used quite extensively even on local file systems to prevent sharing violations. The search indexer is a good example of this. On XP, if indexer was indexing a file and you tried to open the file for write, you’d likely get back STATUS_SHARING_VIOLATION. On Windows 7+, the search indexer takes an oplock on the file; when you open the file for write, the indexer gets notified and it can close its handle which allows your create to succeed.

Oplock Types

First of all, there are enough (8!) different kinds of oplocks at this point so it makes sense to have a cheat sheet. In practice, some are duplicates.  From https://docs.microsoft.com/en-us/windows-hardware/drivers/ifs/overview :

Legacy oplocks:

  • Level 2 (or shared) oplock indicates that there are multiple readers of a stream and no writers. This supports client read caching. [Same as R]

  • Level 1 (or exclusive) oplock allows a client to open a stream for exclusive access and allows the client to perform arbitrary buffering. This supports client read caching and write caching. [Same as RW]

  • Batch oplock (also exclusive) allows a client to keep a stream open on the server even though the local accessor on the client machine has closed the stream. This supports scenarios where the client needs to repeatedly open and close the same file, such as during batch script execution. This supports client read caching, write caching, and handle caching. [Same as RWH]

  • Filter oplock (also exclusive) allows applications and file system filters (including minifilters), which open and read stream data, a way to “back out” when other applications, clients, or both try to access the same stream. This supports client read caching and write caching.

Windows 7 oplocks:

  • Read (R) oplock (shared) indicates that there are multiple readers of a stream and no writers. This supports client read caching. [Same as Level 2]
  • Read-Handle (RH) oplock (shared) indicates that there are multiple readers of a stream, no writers, and that a client can keep a stream open on the server even though the local accessor on the client machine has closed the stream. This supports client read caching and handle caching.
  • Read-Write (RW) oplock (exclusive) allows a client to open a stream for exclusive access and allows the client to perform arbitrary buffering. This supports client read caching and write caching. [Same as level 1]
  • Read-Write-Handle (RWH) oplock (exclusive) allows a client to keep a stream open on the server even though the local accessor on the client machine has closed the stream. This supports client read caching, write caching, and handle caching. [Same as Batch]

Breaking an Oplock

Once one has an oplock, there are a number of file operations that will cause the oplock to be broken (or reduced). Best to just head to https://docs.microsoft.com/en-us/windows-hardware/drivers/ifs/breaking-oplocks for that level of detail, but in general, the following operations can affect an oplock:

  • CREATE
  • READ
  • WRITE
  • CLEANUP
  • Bye range locks

Additionally some file info classes and FSCTLs can lead to oplock breaks.

Oplock-Related Flags in Zw\Flt\IoCreateFile and the Like

When an oplock is broken, the default behavior is to pend the breaking thread until the oplock break has been acknowledged by the owner, or the owner closes the file, or a configurable timeout occurs.

Over the years, some additional flags have been added to CREATE to make this process a little less deadlock-prone.

FILE_COMPLETE_IF_OPLOCKED

If FILE_COMPLETE_IF_OPLOCKED is passed into CREATE, the oplock break is still sent but control returns to the caller with STATUS_OPLOCK_BREAK_IN_PROGRESS. There’s no guarantee as to when the acknowledgement will be processed and so the caller must use some asynchronous mechanism to proceed. More info here: http://www.kasardia.com/some-thoughts-on-oplocks/

This is also a way to detect if a file is oplocked in some cases, though the act of checking can trigger a break. In this scenario, one could just close the HANDLE and be done with it if STATUS_OPLOCK_BREAK_IN_PROGRESS is returned.

FILE_OPEN_REQUIRING_OPLOCK

FILE_OPEN_REQUIRING_OPLOCK is an atomic way to open a file and grab an oplock at the same time. You’re supposed to then request the oplock so you get told about oplock breaks, but it’s not required. From https://docs.microsoft.com/en-us/windows/desktop/api/winternl/nf-winternl-ntcreatefile :

An application that uses this flag must request an oplock after this call succeeds, or all subsequent attempts to open the file will be blocked without the benefit of normal oplock processing. Similarly, if this call succeeds but the subsequent oplock request fails, an application that uses this flag must close its handle after it detects that the oplock request has failed.

If the file is already oplocked, the operation will fail with STATUS_CANNOT_BREAK_OPLOCK and the original oplock remains intact. However, there is some subtlety here:

  1. FILE_OPEN_REQUIRING_OPLOCK represents a RH oplock in Wondows 8+ and an RWH oplock in Windows 7.This means that on Windows 8+, the CREATE will still succeed if someone else has an R or RH oplock. As such, this is not a good way to determine if a file has read oplocks on it, but you could infer that someone has W, WH, or presumably filter oplocks.
  2. Given the previous point, the quote above is not accurate. Not “all subsequent attempts to open the file’ will be blocked. Just ones that conflict with RH oplocks.

Available on Windows 7+.

FILE_RESERVE_OPFILTER

Nothing to add for now. I don’t see filter oplocks very much, perhaps because they lack the fine grained control of the Windows 7 oplocks.

Testing Oplocks

As usual, Zezula’s FileTest tool is the easiest way to test oplock behavior. You can find it here: http://www.zezula.net/en/fstools/filetest.html

oplock1

Debugging Oplocks

In Windows 8+, the !fltkd.oplock extension in WinDBG can be used to figure out who owns an oplock.

Additional Resources

I was surprised to see that even Wikileaks has a page on oplocks: https://wikileaks.org/ciav7p1/cms/page_38633506.html

Advertisements
Posted in File System, Windows | Leave a comment

Tracking Down a FILE_OBJECT leak

I recently looked at an an issue where a kernel component was leaking FILE_OBJECTs. The handle count on the objects was zero but they all had outstanding references keeping them open.

Note: This is not new territory but I had to fumble around a lot. There’s a similar theme here on NTFSD: FileObjects in system Dump.

Symptoms

  • We had a machine that was crashing every day and almost always in a different path.
    kd> vertarget
    Windows 7 Kernel Version 7601 (Service Pack 1) UP Free x86 compatible
    Product: WinNt, suite: TerminalServer SingleUserTS
    Built by: 7601.24335.x86fre.win7sp1_ldr_escrow.181228-0954
    Machine Name:
    Kernel base = 0x82c3e000 PsLoadedModuleList = 0x82d93730
    Debug session time: Wed Jan 23 20:47:12.740 2019 (UTC - 8:00)
    System Uptime: 0 days 16:05:38.018
  • At the time of the crash, there was always hard evidence of memory exhaustion.
    kd> !vm
    Page File: \??\C:\pagefile.sys
    Current: 3452408 Kb Free Space: 3309408 Kb
    Minimum: 3452408 Kb Maximum: 9435624 Kb
    
    Physical Memory: 786302 ( 3145208 Kb)
    Available Pages: 235238 ( 940952 Kb)
    ResAvail Pages: 472294 ( 1889176 Kb)
    Locked IO Pages: 0 ( 0 Kb)
    Free System PTEs: 1952 ( 7808 Kb)
    
    ******* 9158 system cache map requests have failed ******
    
    Modified Pages: 14261 ( 57044 Kb)
    Modified PF Pages: 14260 ( 57040 Kb)
    Modified No Write Pages: 1 ( 4 Kb)
    NonPagedPool Usage: 222941 ( 891764 Kb)
    NonPagedPool Max: 523004 ( 2092016 Kb)
    PagedPool 0: 24571 ( 98284 Kb)
    PagedPool 1: 22041 ( 88164 Kb)
    PagedPool 2: 19187 ( 76748 Kb)
    PagedPool 3: 19132 ( 76528 Kb)
    PagedPool 4: 19232 ( 76928 Kb)
    PagedPool Usage: 104163 ( 416652 Kb)
    PagedPool Maximum: 523264 ( 2093056 Kb)
    
    ********** 2536056 pool allocations have failed **********
  • There were hundreds of thousands outstanding allocations for file objects and associated filter manager structures.
    kd> !poolused 0x3
    .
    Sorting by NonPaged Pool Consumed
    
    NonPaged Paged
    Tag Allocs Frees Diff Used Allocs Frees Diff Used
    
    File 40342160 40092340 249820 49940032 0 0 0 0 File objects 
    FMfc 415334 170461 244873 27425776 0 0 0 0 FLTMGR_FILE_OBJECT_CONTEXT structure , Binary: fltmgr.sys
  • The process handle counts were pretty low (no process had more than a few thousand handles open). Keep in mind that every successful call to CreateFile results in a new file object. Usually these file objects are cleaned up and closed when the last handle is closed. Given the low handle count, this pointed to a kernel component leaving an outstanding reference on the file objects.

Diagnosing

Since the ‘File’ pooltag is only used for file objects, it was pretty straightforward to see that there were hundreds of thousands of open files on the system. I let !poolfind run to completion:

.logopen c:\temp\poolfind_file.txt
Opened log file 'c:\temp\poolfind_file.txt'
kd> !poolfind File

Scanning large pool allocation table for tag 0x656c6946 (File) (a4200000 : a4400000)

87adbb50 : tag File (Protected), size 0xc0, Nonpaged pool
954f6a68 : tag File (Protected), size 0xc0, Nonpaged pool
95da8f40 : tag File (Protected), size 0xc0, Nonpaged pool
8ae0fed8 : tag File (Protected), size 0xb0, Nonpaged pool
a55d14a8 : tag File (Protected), size 0xc0, Nonpaged pool
a55d1c88 : tag File (Protected), size 0xc0, Nonpaged pool
...

.logclose

Next, I wanted to see the names of the files to look for patterns. That proved to be largely unhelpful and messy but worth documenting (please comment if you have a better way to do this). The legacy WinDBG scripting language supports looping on file data but I had to sanitize the input first (I couldn’t figure out how to read a file using Javascript).

  1. I used Sublime Text to do a regular expression Find/Replace so that each line only had a single hex address.
    Find: ([0-9a-f]+).* 
    Replace: \1
  2. !poolfind gives the address of the pooltag itself so I had to add an offset to the address to get past the rest of the _POOL_HEADER and all of the _OBJECT_HEADER. In this case, the offset was 0n64 bytes but I could not figure out how to script this. So, I copied the entire list into Excel and used the HEX2DEC(x) + 64 formula to give me a list of addresses that I wanted. I didn’t bother converting back to HEX because dx assumes numbers are in base 10.
  3. Now, I was able to run the following script to get the name of each FileObject
    .foreach /f ( test "c:\temp\file.txt") { dx ((_FILE_OBJECT*) ${test})->FileName }
    ((_FILE_OBJECT*) 2276309904)->FileName                  : "\Device\HarddiskVolume1\Windows\ServiceProfiles\NetworkService\NTUSER.DAT" [Type: _UNICODE_STRING]
        [<Raw View>]     [Type: _UNICODE_STRING]
    ((_FILE_OBJECT*) 2505009832)->FileName                  : "\ProgramData\Contoso\Debug.log" [Type: _UNICODE_STRING]
        [<Raw View>]     [Type: _UNICODE_STRING]
    ((_FILE_OBJECT*) 2514128768)->FileName                  : "\Windows\System32\rsaenh.dll" [Type: _UNICODE_STRING]
        [<Raw View>]     [Type: _UNICODE_STRING]
    ((_FILE_OBJECT*) 2330001176)->FileName                  : "" [Type: _UNICODE_STRING]
        [<Raw View>]     [Type: _UNICODE_STRING]
    ((_FILE_OBJECT*) 2774340840)->FileName                  : "\ProgramData\Contoso\Debug.log" [Type: _UNICODE_STRING]
        [<Raw View>]     [Type: _UNICODE_STRING]
    ((_FILE_OBJECT*) 2774342856)->FileName                  : "\Windows\System32\framedynos.dll" [Type: _UNICODE_STRING]
    ...

In this case, the filenames didn’t narrow things down much but it might come in handy another time. As one might expect, some files had hundreds (and possibly thousands?) of open file objects.

Object Tracing

Time for the big guns. Using glfags, you can enable object tracing for a particular pooltag without even rebooting!

Note: If you want the setting to persist after reboot, also add it to the “System Registry” tab.

gflags_objecttracing

After running for a few hours, we had a new BSOD. We tracked down one of the leaking file objects and were able to see the stacks for each call to ObReference* and ObDereference* using !obtrace <file object>! It appears that the contoso driver is not calling ObDereferenceObject on the file object returned from FltCreateFileEx2.

kd> !object d0df4f80
Object: d0df4f80 Type: (851cf3f8) File
ObjectHeader: d0df4f68 (new version)
HandleCount: 0 PointerCount: 1
Directory Object: 00000000 Name: \PROGRA~2\Contoso\Logs\contoso.l {HarddiskVolume1}

kd> !obtrace d0df4f80
Object: d0df4f80
Image: System
Sequence (+/-) Tag Stack
-------- ----- ---- ---------------------------------------------------
d3ce41 +1 Dflt nt!ObCreateObject+1c5 
nt!IopAllocRealFileObject+50 
nt!IopParseDevice+af8 
nt!ObpLookupObjectName+510 
nt!ObOpenObjectByName+165 
nt!IopCreateFile+6c3 
nt!IoCreateFileEx+9e 
fltmgr!FltpCreateFile+ba 
fltmgr!FltCreateFileEx2+5a 
contoso+20a8 
contoso+10a37b 
contoso+7a785

d3ce42 +1 Dflt nt!ObfReferenceObjectWithTag+27 
nt!ObfReferenceObject+12 
nt!IopParseDevice+13c9 
nt!ObpLookupObjectName+510 
nt!ObOpenObjectByName+165 
nt!IopCreateFile+6c3 
nt!IoCreateFileEx+9e 
fltmgr!FltpCreateFile+ba 
fltmgr!FltCreateFileEx2+5a 
contoso+20a8 
contoso+10a37b 
contoso+7a785

d3ce43 -1 Dflt nt!ObfDereferenceObjectWithTag+22 
nt!IoCreateFileEx+9e 
fltmgr!FltpCreateFile+ba 
fltmgr!FltCreateFileEx2+5a 
contoso+20a8 
contoso+10a37b 
contoso+7a785 
contoso+857d 
contoso+8b37 
contoso+4138 
nt!PspSystemThreadStartup+159

d3ce44 +1 Dflt nt!ObReferenceObjectByHandleWithTag+254 
nt!ObReferenceObjectByHandle+21 
fltmgr!FltpCreateFile+dc 
fltmgr!FltCreateFileEx2+5a 
contoso+20a8 
contoso+10a37b 
contoso+7a785 
contoso+857d 
contoso+8b37 
contoso+4138 
nt!PspSystemThreadStartup+159

d3ce45 +1 Dflt nt!ObReferenceFileObjectForWrite+155 
nt!NtWriteFile+3f 
nt!KiSystemServicePostCall+0 
nt!ZwWriteFile+11 
contoso+4156 
nt!PspSystemThreadStartup+159

d3ce46 +1 Dflt nt!ObfReferenceObjectWithTag+27 
nt!ObfReferenceObject+12 
nt!NtWriteFile+25a 
nt!KiSystemServicePostCall+0 
nt!ZwWriteFile+11 
contoso+4156 
nt!PspSystemThreadStartup+159

d3ce47 -1 Dflt nt!ObDereferenceObjectDeferDeleteWithTag+29 
nt!ObDereferenceObjectDeferDelete+13 
nt!IopCompleteRequest+433 
nt!IopfCompleteRequest+3b4 
nt!IovCompleteRequest+133 
Ntfs!NtfsExtendedCompleteRequestInternal+107 
Ntfs!NtfsCommonWrite+25cd 
Ntfs!NtfsFsdWrite+2e1 
nt!IovCallDriver+258 
nt!IofCallDriver+1b 
fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+2b0 
fltmgr!FltpDispatch+c5 
nt!IovCallDriver+258 
nt!IofCallDriver+1b 
nt!IopSynchronousServiceTail+1f8 
nt!NtWriteFile+6f8

d3ce48 -1 Dflt nt!ObfDereferenceObjectWithTag+22 
nt!NtWriteFile+6f8 
nt!KiSystemServicePostCall+0 
nt!ZwWriteFile+11 
contoso+4156 
nt!PspSystemThreadStartup+159

d3ce49 +1 Dflt nt!ObfReferenceObjectWithTag+27 
nt!ObfReferenceObject+12 
nt!IopCloseFile+23e 
nt!ObpDecrementHandleCount+139 
nt!ObpCloseHandleTableEntry+203 
nt!ObpCloseHandle+7f 
nt!NtClose+4e 
nt!KiSystemServicePostCall+0 
nt!ZwClose+11 
contoso+415f 
nt!PspSystemThreadStartup+159

d3ce4a -1 Dflt nt!ObfDereferenceObjectWithTag+22 
nt!ObpDecrementHandleCount+139 
nt!ObpCloseHandleTableEntry+203 
nt!ObpCloseHandle+7f 
nt!NtClose+4e 
nt!KiSystemServicePostCall+0 
nt!ZwClose+11 
contoso+415f 
nt!PspSystemThreadStartup+159

d3ce4b -1 Dflt nt!ObfDereferenceObjectWithTag+22 
nt!ObpCloseHandle+7f 
nt!NtClose+4e 
nt!KiSystemServicePostCall+0 
nt!ZwClose+11 
contoso+415f 
nt!PspSystemThreadStartup+159

-------- ----- ---------------------------------------------------
References: 6, Dereferences 5

Note: Some file objects were not in the object tracing database for some reason. To make things easier, the MEX !obtrace extension will actually spit out the addresses of all the object headers in the tracing database.

kd> !mex.obtrace 1
Mex External 3.0.0.7172 Loaded!
Debug: Working on _OBJECT_REF_INFO fffffffffd509000, ObjectHeader: fffffffff06ae828, NextRef: fffffffffd422000, ImageFileName: contoso.exe, ACTUAL proc name:
Debug: Working on _OBJECT_REF_INFO fffffffffd422000, ObjectHeader: fffffffff0be3860, NextRef: fffffffffd183000, ImageFileName: System, ACTUAL proc name:
Debug: Working on _OBJECT_REF_INFO fffffffffd183000, ObjectHeader: fffffffff4ec0998, NextRef: fffffffffa9cf000, ImageFileName: contosoe.exe, ACTUAL proc name:
Debug: Working on _OBJECT_REF_INFO fffffffffa9cf000, ObjectHeader: ffffffffbf3d4908, NextRef: fffffffffcb7a000, ImageFileName: contoso.exe, ACTUAL proc name:
Debug: Working on _OBJECT_REF_INFO fffffffffcb7a000, ObjectHeader: ffffffffcf7e7ca8, NextRef: ffffffffefcdb000, ImageFileName: scsrvc.exe, ACTUAL proc name: 
Debug: Working on _OBJECT_REF_INFO ffffffffefcdb000, ObjectHeader: ffffffff9c56ef68, NextRef: fffffffffb7df000, ImageFileName: System, ACTUAL proc name:

...
Posted in Debugging, File System, GFlags, Windbg, Windows | Leave a comment

64-bit HANDLEs can be truncated to 32-bits

I came across this bit of code thanks to a compiler warning (obviously, error checking has been removed):

HANDLE handle = CreateFileW(…)
INT lzHandle = LzInit((INT) handle);

On 64-bit Windows, this results in a truncation of ‘handle’, but it turns out that this is not a programming error. It’s actually a common scenario, used when passing HANDLEs between 32-bit and 64-bit processes.

From MSDN:

64-bit versions of Windows use 32-bit handles for interoperability. When sharing a handle between 32-bit and 64-bit applications, only the lower 32 bits are significant, so it is safe to truncate the handle (when passing it from 64-bit to 32-bit) or sign-extend the handle (when passing it from 32-bit to 64-bit). Handles that can be shared include handles to user objects such as windows (HWND), handles to GDI objects such as pens and brushes (HBRUSH and HPEN), and handles to named objects such as mutexes, semaphores, and file handles.

I suspect I used to know this. Noting it for future reference.

Posted in SDL, Windows | Leave a comment

IO_IGNORE_SHARE_ACCESS_CHECK

Update (May 2019)

The documentation for FltCreateFileEx2 has been updated to be less ambiguous. Thanks MS!

Options flags Meaning
IO_IGNORE_SHARE_ACCESS_CHECK Indicates that the I/O manager should not perform share-access checks on the file object after it is created. However, the file system might still perform these checks. Note that when IO_IGNORE_SHARE_ACCESS_CHECK is specified, the file system does not track the current open’s desired access or shared access. Because of this, subsequent open calls on the same file may succeed.

Documented Behavior

The IO_IGNORE_SHARE_ACCESS_CHECK flag provides a way for Windows kernel components to bypass sharing checks while opening local files. NTFSD has lots of posts on the subject — see here, here, here, etc…

MSDN discusses the flag as well but the wording is somewhat ambiguous. For example, the documentation for FltCreateFileEx2 mentions IO_IGNORE_SHARE_ACCESS_CHECK in a couple of places:

Options flags Meaning
IO_IGNORE_SHARE_ACCESS_CHECK Indicates that the I/O manager should not perform share-access checks on the file object after it is created. However, the file system might still perform these checks.

And later, a similarly ambiguous note:

Note If IO_IGNORE_SHARE_ACCESS_CHECK is specified in the Flags parameter, the I/O manager ignores the ShareAccess parameter. However, the file system might still perform access checks. Thus, it is important to specify the sharing mode you would like for the ShareAccess parameter, even when using the IO_IGNORE_SHARE_ACCESS_CHECK flag.

I’ve been using this flag for years, and I thought I understood the ambiguity in the documentation.

  1. The documentation fails to mention that this flag has no effect on files opened across the network.
  2. My understanding was that this flag *might* not be honored on third-party file systems since they may perform sharing access checks without the help of the IO manager.

I was slightly off on the second point — it turns out that even MS file systems perform internal sharing checks in some cases — see the Bonus section at the end.

On top of this, using IO_IGNORE_SHARE_ACCESS_CHECK results in outright undocumented behavior when subsequently opening the same file.

Note: I am testing/disassembling on Windows 10 RS3.

Undocumented Behavior

On a couple of occasions, we’ve been surprised at the outcome of the following test:

  1. Open an existing file via FltCreateFileEx2 (or similar) with:
    • DesiredAccess = FILE_READ_DATA
    • ShareAccess = FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE
    • Flags = IO_IGNORE_SHARE_ACCESS_CHECK
  2. While the file handle is still open, open the same file with:
    • DesiredAccess = FILE_WRITE_DATA
    • ShareAccess = 0
    • Flags = 0

We expect the second open to fail with STATUS_SHARING_VIOLATION, since the file is already open for READ and the sharing flags are 0.

However, the second open succeeds. We find the same regardless of the desired access of the second open — FILE_READ_DATA and DELETE also work. We’ve reproduced this on Windows 7 SP1 as well as Windows 10 SP3.

When I finally decided to post my puzzlement to NTFSD, the site was down due to an OSR office move. So, I dove in. I later found someone else asking about this behavior but I did not come across an explanation for it in the archives.

IoCheckShareAccess

The IO Manager provides a set of functions to make it easier for file systems to handle share access checking. In particular, IoCheckShareAccess and IoCheckShareAccessEx can be used to check share access permissions when opening a file.

Local Microsoft file systems use these functions (you can see this by examining fastfat!FatCheckShareAccess or by disassembling NTFS!NtfsCheckShareAccess). Here’s the signature for IoCheckShareAccess:

 NTKERNELAPI NTSTATUS IoCheckShareAccess(
  ACCESS_MASK   DesiredAccess,
  ULONG         DesiredShareAccess,
  PFILE_OBJECT  FileObject,
  PSHARE_ACCESS ShareAccess,
  BOOLEAN       Update
);

MSDN explains the parameters and touches on the relevance of ShareAccess — this is a struct that is stored in the File/Directory/Stream Control Block (FCB/DCB/SCB) of a file object and is therefore common to all open instances of the file/directory/stream. Here’s the definition from a recent WDK:

typedef struct _SHARE_ACCESS {
    ULONG OpenCount;
    ULONG Readers;
    ULONG Writers;
    ULONG Deleters;
    ULONG SharedRead;
    ULONG SharedWrite;
    ULONG SharedDelete;
} SHARE_ACCESS, *PSHARE_ACCESS;

And, here’s a description of what each field should contain:

OpenCount The number of open file objects with (FILE_READ_DATA | FILE_EXECUTE | FILE_WRITE_DATA | FILE_APPEND_DATA | DELETE) access (for the same file/directory/stream)
Readers The number of open file objects with (FILE_READ_DATA | FILE_EXECUTE) access
Writers The number of open file objects with (FILE_WRITE_DATA | FILE_APPEND_DATA) access
Deleters The number of open file objects with DELETE access
SharedRead The number of open file objects with FILE_SHARE_READ
SharedWrite The number of open file objects with FILE_SHARE_WRITE
SharedDelete The number of open file objects with FILE_SHARE_DELETE

IoCheckShareAccess is not a huge function and so I disassembled it to see what it does (actually, I disassembled IoCheckLinkShareAccess since that’s the underlying function). IoCheckLinkAccess does the following:

  1. Set the ReadAccess, WriteAccess, and DeleteAccess bits in the FILE_OBJECT, based on the DesiredAccess parameter. If all of these are 0, return with STATUS_SUCCESS.
  2. Set the SharedRead, SharedWrite, and SharedDelete bits in the FILE_OBJECT based on the ShareAccess parameter. At this point, if the IO_IGNORE_SHARE_ACCESS_CHECK flag is present, return with STATUS_SUCCESS.
  3. Check that DesiredAccess does not collide with the share access of existing opens. If there is a collision, return STATUS_SHARING_VIOLATION.
  4. Check that DesiredSharedAccess does not collide with the access permissions of existing opens. If there is a collision, return STATUS_SHARING_VIOLATION.
  5. Finally, if the share access check passes and Update is non-zero, update the relevant counts in the common ShareAccess struct.

The key piece — if IO_IGNORE_SHARE_ACCESS_CHECK is passed in, IoCheckShareAccess skips steps 3-5. This means that the share access check will always pass for the current open, which is what we expect. Additionally, since the ShareAccess struct is not updated, share access checks for subsequent opens will not be affected by the current open!

Final Thoughts

The implementation of IO_IGNORE_SHARE_ACCESS_CHECK makes a lot of sense when you consider concurrency in the file system. After all, a call to FltCreateFileEx2 with IO_IGNORE_SHARE_ACCESS_CHECK and a a call to CreateFile can be issued at the same time. It would be horrendously frustrating if the CreateFile sometimes succeeded and sometimes failed, because of some inconsiderate kernel driver.

The documentation needs to be updated to make this behavior clear. I’ve submitted a doc issue to FltCreateFileEx2 via GitHub issues (this seems new).

Big thanks to MS for making the fastfat source available.

Bonus #1

Here’s another reason why the MSDN docs are ambiguous — even local Microsoft file systems can trigger a sharing violation when the IO_IGNORE_SHARE_ACCESS_CHECK flag is passed in.

When a file system sees IRP_MJ_CLEANUP for a file object, the counters in the associated _SHARE_ACCESS struct are decremented via IoRemoveShareAccess. However, the file object can still be used for memory-mapped IO and so the file system may perform additional sharing checks.

In particular, the fastfat FatCheckShareAccess function checks for writable sections before calling IoCheckShareAccess. Here’s the excerpt:

#if (NTDDI_VERSION >= NTDDI_VISTA)
    //
    //  Do an extra test for writeable user sections if the user did not allow
    //  write sharing - this is neccessary since a section may exist with no handles
    //  open to the file its based against.
    //

    if ((NodeType( FcbOrDcb ) == FAT_NTC_FCB) &&
        !FlagOn( ShareAccess, FILE_SHARE_WRITE ) &&
        FlagOn( *DesiredAccess, FILE_EXECUTE | FILE_READ_DATA | FILE_WRITE_DATA | FILE_APPEND_DATA | DELETE | MAXIMUM_ALLOWED ) &&
        MmDoesFileHaveUserWritableReferences( &FcbOrDcb->NonPaged->SectionObjectPointers )) {

        return STATUS_SHARING_VIOLATION;
    }
#endif

It is reasonable to assume that NTFS does the same thing, but on a cursory disassembly of NTFS!NtfsCheckShareAccess I didn’t see any calls to MmDoesFileHaveUserWriteReferences.

Bonus #2 – Where is IO_IGNORE_SHARE_ACCESS_CHECK stored?

This is just a bit of trivia really; it appears that if you need to enable this behavior without the use of the IO_IGNORE_SHARE_ACCESS_CHECK flag, a minifilter can just call IoSetFileObjectIgnoreSharing on Windows 7+. However…

At present, when the IO Manager sees the IO_IGNORE_SHARE_ACCESS_CHECK flag, it sets bit 0 in the _FILE_OBJECT->FileObjectExtension.FoExtFlags field.

nt!_FILE_OBJECT
   ...
   +0x04a ReadAccess       : UChar
   +0x04b WriteAccess      : UChar
   +0x04c DeleteAccess     : UChar
   +0x04d SharedRead       : UChar
   +0x04e SharedWrite      : UChar
   +0x04f SharedDelete     : UChar
   ...
   +0x0d0 FileObjectExtension : Ptr64 Void

nt!_IOP_FILE_OBJECT_EXTENSION
+ 0x000 FoExtFlags : Uint4B
+ 0x008 FoExtPerTypeExtension : [9] Ptr64 Void
+ 0x050 FoIoPriorityHint : _IOP_PRIORITY_HINT

We can see this by looking at the disassembly for IoIsFileObjectIgnoringSharing, which “determines if a file object is set with the option to ignore file sharing access checks.”.

0: kd> uf IoIsFileObjectIgnoringSharing
nt!IoIsFileObjectIgnoringSharing:
// rax = FileObject->FileObjectExtension
fffff803`38865520 488b81d0000000  mov     rax,qword ptr [rcx+0D0h]
fffff803`38865527 4885c0          test    rax,rax
fffff803`3886552a 7505            jne     nt!IoIsFileObjectIgnoringSharing+0x11 (fffff803`38865531)  Branch

nt!IoIsFileObjectIgnoringSharing+0xc:
fffff803`3886552c 32c9            xor     cl,cl

nt!IoIsFileObjectIgnoringSharing+0xe:
fffff803`3886552e 8ac1            mov     al,cl
fffff803`38865530 c3              ret

nt!IoIsFileObjectIgnoringSharing+0x11:
// eax = FileObject->FileObjectExtension->FoExtFlags
fffff803`38865531 8b00            mov     eax,dword ptr [rax]
fffff803`38865533 b101            mov     cl,1
// test 1, (FileObject->FileObjectExtension->FoExtFlags & 0xf)
fffff803`38865535 84c1            test    cl,al
fffff803`38865537 74f3            je      nt!IoIsFileObjectIgnoringSharing+0xc (fffff803`3886552c)  Branch

nt!IoIsFileObjectIgnoringSharing+0x19:
fffff803`38865539 ebf3            jmp     nt!IoIsFileObjectIgnoringSharing+0xe (fffff803`3886552e)  Branch
Posted in File System, Reversing, Windows | Leave a comment

Finding an Exception in a user-mode minidump

I spend most of my time in kernel and so I still fumble around sometimes when asked to look at user-mode crash dumps. In particular, someone gave me an .hdmp file recently — I don’t know much about this kind of dump but it appears to be a mini-dump with some additional heap info in it.

When I opened the file, WinDBG had some bad news:

ERROR: Unable to find system thread 3DC
ERROR: The thread being debugged has either exited or cannot be accessed
ERROR: Many commands will not work properly
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
ERROR: Exception C0000005 occurred on unknown thread 3DC
(1150.3dc): Access violation - code c0000005 (first/second chance not available)

Oh well. At least .excr works.

0:???> .excr
WARNING: The debugger does not have a current process or thread
WARNING: Many commands will not work
WARNING: The debugger does not have a current process or thread
WARNING: Many commands will not work

What! WinDBG, why did you lie to me?

After messing around for a bit and accidentally fixing the problem (by running !analyze -v), I realized that WinDBG did give me a hint of what was wrong. I tried resetting the current process and things started magically working.

0:???> |s 0
WARNING: The debugger does not have a current process or thread
WARNING: Many commands will not work
ntdll!NtDeviceIoControlFile+0x14:
00007ffc`83edff64 c3 ret
 ^ Syntax error in '|s 0'

0:000> .excr
rax=0000000000000000 rbx=000000c25ddfef48 rcx=000000c25ddfed50
rdx=000000c25ddfed40 rsi=00007ff60db99dc0 rdi=000001c6f3fd4870
rip=0000000000000000 rsp=000000c25ddfecb8 rbp=000000c25ddfed69
 r8=0000000000000001 r9=00007ff60dad3d38 r10=0000000000000000
r11=000000c25ddfec20 r12=00007ff60dae4b80 r13=0000000000000000
r14=00007ff60dad6030 r15=000001c6f3fd4870
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
00000000`00000000 ?? ???

 

 

Posted in Debugging, usermode, Windbg | Leave a comment

Different Ways to View Assembly Code

tl;dr; Compiling with /FAcs gives you useful source + assembler output.

There are lots of ways to view the compiled assembly code for a function — in this post I discuss three of them. I work exclusively on Windows so your mileage will vary on other platforms. I suppose if I had an IDA Pro license, I’d just use that.

Here is my simple program. I’m interested in seeing the assembler version of DoSomething.

#include 

void DoSomething()
{
  wprintf(L"Hello world!");
}

int wmain(int argc, wchar_t* argv[])
{
  DoSomething();
  return 0;
}

View Assembly Code Using the Debugger

Generally, if I want to disassemble something, its because I’m already in the debugger (WinDBG). For this to work, I need to have symbols available.

Here’s the debug version of DoSomething:

0:000> uf dosomething
Sandbox!DoSomething [c:\src\sandbox\sandbox.cpp @ 13]:
 13 004138f0 55 push ebp
 13 004138f1 8bec mov ebp,esp
 13 004138f3 81ecc0000000 sub esp,0C0h
 13 004138f9 53 push ebx
 13 004138fa 56 push esi
 13 004138fb 57 push edi
 13 004138fc 8dbd40ffffff lea edi,[ebp-0C0h]
 13 00413902 b930000000 mov ecx,30h
 13 00413907 b8cccccccc mov eax,0CCCCCCCCh
 13 0041390c f3ab rep stos dword ptr es:[edi]
 14 0041390e 68b0ed4100 push offset Sandbox!`string' (0041edb0)
 14 00413913 e848dbffff call Sandbox!ILT+1115(_wprintf) (00411460)
 14 00413918 83c404 add esp,4
 15 0041391b 5f pop edi
 15 0041391c 5e pop esi
 15 0041391d 5b pop ebx
 15 0041391e 81c4c0000000 add esp,0C0h
 15 00413924 3bec cmp ebp,esp
 15 00413926 e805d9ffff call Sandbox!ILT+555(__RTC_CheckEsp) (00411230)
 15 0041392b 8be5 mov esp,ebp
 15 0041392d 5d pop ebp
 15 0041392e c3 ret

Here’s what I see for the release version (DoSomething has been inlined into wmain)

0:000> uf sandbox!dosomething
Sandbox!wmain [c:\src\sandbox\sandbox.cpp @ 12]:
   12 00c11060 68ac21c100      push    offset Sandbox!`string' (00c121ac)
   13 00c11065 e8c6ffffff      call    Sandbox!wprintf (00c11030)
   13 00c1106a 83c404          add     esp,4
   14 00c1106d 33c0            xor     eax,eax
   15 00c1106f c3              ret

There are reasons why using the debugger is not always a good option:

  1. I can’t run the code in my debug environment.
  2. It can take a long time to set up a debug environment and I really just want to check something simple.
  3. The function has been inlined and is not available as an independent entity.
  4. I don’t have symbols.
  5. Depending on the compiler optimizations, the debugger can get confused.
  6. etc…

View Assembly Code Using Dumpbin

If I have object files, I can use dumpbin to extract the assembly version of the function. The command is like so:

c:\src\sandbox>dumpbin /disasm Sandbox.obj > dumpbin.txt

Note: This won’t help much for release executable because symbols have been stripped. But.. it can work on static libraries, since they are zipped archives that contain object files.

Here’s the debug output for DoSomething and is fairly similar to the WinDBG output:

?DoSomething@@YAXXZ (void __cdecl DoSomething(void)):
 00000000: 55 push ebp
 00000001: 8B EC mov ebp,esp
 00000003: 81 EC C0 00 00 00 sub esp,0C0h
 00000009: 53 push ebx
 0000000A: 56 push esi
 0000000B: 57 push edi
 0000000C: 8D BD 40 FF FF FF lea edi,[ebp-0C0h]
 00000012: B9 30 00 00 00 mov ecx,30h
 00000017: B8 CC CC CC CC mov eax,0CCCCCCCCh
 0000001C: F3 AB rep stos dword ptr es:[edi]
 0000001E: 68 00 00 00 00 push offset ??_C@_1BK@IHDEKFOI@?$AAH?$AAe?$AAl?$AAl?$AAo?$AA?5?$AAw?$AAo?$AAr?$AAl?$AAd?$AA?$CB?$AA?$AA@
 00000023: E8 00 00 00 00 call _wprintf
 00000028: 83 C4 04 add esp,4
 0000002B: 5F pop edi
 0000002C: 5E pop esi
 0000002D: 5B pop ebx
 0000002E: 81 C4 C0 00 00 00 add esp,0C0h
 00000034: 3B EC cmp ebp,esp
 00000036: E8 00 00 00 00 call __RTC_CheckEsp
 0000003B: 8B E5 mov esp,ebp
 0000003D: 5D pop ebp
 0000003E: C3 ret

Here’s the release version:

Dump of file Sandbox.obj

File Type: ANONYMOUS OBJECT

Uh-oh. Dumpbin can’t help me as I’ve turned on whole program optimization.  The docs have this to say:

Only the /HEADERS DUMPBIN option is available for use on files produced by the /GL (Whole program optimization) compiler option.+

Back to WinDBG output? Nope, one more option…

View Assembly Code Using /FAcs Compiler Flags

If I were thinking ahead, or if I am able to rebuild from source, my colleague pointed out that you can generate very nice source + assembly at compile time using the /FAcs compiler flag.

This option will result in an extra file with a .cod suffix. The file contains machine code, assembly code, and source code! Clearly, this is the most verbose and quickest to decipher option.

In Visual Studio 2017, this option is under Properites -> C/C++ -> Output Files -> Assembler Output.

facs_compiler_option

Here’s the debug version:

; Function compile flags: /Odtp /RTCsu /ZI
; File c:\src\sandbox\sandbox.cpp
; COMDAT ?DoSomething@@YAXXZ
_TEXT SEGMENT
?DoSomething@@YAXXZ PROC ; DoSomething, COMDAT

; 13 : {

00000 55 push ebp
 00001 8b ec mov ebp, esp
 00003 81 ec c0 00 00
 00 sub esp, 192 ; 000000c0H
 00009 53 push ebx
 0000a 56 push esi
 0000b 57 push edi
 0000c 8d bd 40 ff ff
 ff lea edi, DWORD PTR [ebp-192]
 00012 b9 30 00 00 00 mov ecx, 48 ; 00000030H
 00017 b8 cc cc cc cc mov eax, -858993460 ; ccccccccH
 0001c f3 ab rep stosd

; 14 : wprintf(L"Hello world!");

0001e 68 00 00 00 00 push OFFSET ??_C@_1BK@IHDEKFOI@?$AAH?$AAe?$AAl?$AAl?$AAo?$AA?5?$AAw?$AAo?$AAr?$AAl?$AAd?$AA?$CB?$AA?$AA@
 00023 e8 00 00 00 00 call _wprintf
 00028 83 c4 04 add esp, 4

; 15 : }

0002b 5f pop edi
 0002c 5e pop esi
 0002d 5b pop ebx
 0002e 81 c4 c0 00 00
 00 add esp, 192 ; 000000c0H
 00034 3b ec cmp ebp, esp
 00036 e8 00 00 00 00 call __RTC_CheckEsp
 0003b 8b e5 mov esp, ebp
 0003d 5d pop ebp
 0003e c3 ret 0
?DoSomething@@YAXXZ ENDP ; DoSomething
_TEXT ENDS

Here is the release version:

; Function compile flags: /Ogtp
 ; File c:\src\sandbox\sandbox.cpp
 ; COMDAT ?DoSomething@@YAXXZ
 _TEXT SEGMENT
 ?DoSomething@@YAXXZ PROC ; DoSomething, COMDAT

; 8 : wprintf(L"Hello world!");

00000 68 00 00 00 00 push OFFSET ??_C@_1BK@IHDEKFOI@?$AAH?$AAe?$AAl?$AAl?$AAo?$AA?5?$AAw?$AAo?$AAr?$AAl?$AAd?$AA?$CB?$AA?$AA@
 00005 e8 00 00 00 00 call _wprintf
 0000a 59 pop ecx

; 9 : }

0000b c3 ret 0
 ?DoSomething@@YAXXZ ENDP ; DoSomething
 _TEXT ENDS
Posted in Debugging | Leave a comment

Preventing System DLL Sideload Attacks

Problem Statement

DLL sideloading is an attack where an unintended DLL is loaded, resulting in unintended code execution. This attack is possible for any DLL; in this article, we focus on Windows system DLLs.

Sideloading system DLLs

There’s an inherent problem when loading a Windows system DLL like winhttp.dll. For compatibility reasons, the default Windows DLL search begins in the directory where the executable was launched. An attacker need only copy a rogue version of the system DLL to the same directory as the executable. That’s pretty easy even without elevated privileges — an attacker can copy the executable + the rogue DLL to the temp folder, and execute the binary from there.

This vulnerability is very common when loading Windows system DLLs because we usually omit the full path for these modules.

  • Compile-time dynamically-linked libraries are resolved by DLL name alone before execution begins. This is not something we can change.
  • When loading system DLLs dynamically at runtime, we generally pass just the DLL name and let the loader figure out where the system directory is.
  • We generally skip trust checks on system DLLs since a) we don’t have the full path, and b) we rely on system functions like WinVerifyTrust to validate the signature. If the operating system has been compromised already, the trust check won’t really give us much confidence.

Sideloading is especially effective for executables that run with elevated privileges and are embedded signed. For example, many users will quickly agree to elevate privileges if the UAC prompts tells them that their trusted anti-virus vendor wants to make changes.

This is less of an issue for Microsoft executables since they are usually catalog signed. This means that moving them to another location breaks trust. For example, here’s the dialog when we copy mmc.exe out of the system directory and execute it:

This is not a new attack vector, but it’s worth understanding the nuances and mitigations.

MSDN has a good amount of information about this topic but it is confusing and spread out over multiple articles. When not otherwise specified, the details in this article come from the MSDN article entitled Dynamic-Link Library Search Order. Additional details are provided where appropriate.

Searching for DLLs

By default, when a DLL load is done by name alone, the default DLL search path is:

  • The directory from which the application loaded.
  • The system directory.
  • The 16-bit system directory.
  • The Windows directory.
  • The current directory.
  • The directories that are listed in the PATH environment variable.

There are a couple of exceptions:

  1. If a DLL with the same name is already loaded in the process, the existing DLL is used.
  2. If the DLL is in the list of Known DLLs, it has already been loaded at boot time and so the existing DLL is used.The list of known DLLs is taken from HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\KnownDLLs and is then augmented by additional DLLs that are statically linked to the DLLs in the list. For more information, see Larry Osterman’s article entitled What are Known DLLs anyway?.

    If you use WinObj to view the Object directory, you can see the full list of Known DLLs on a system (and yes, the list of Known DLLs is an attack vector but it is out of scope for this article). Note that this list varies by operating system. For example, on Windows Vista, crypt32.dll is not in the Known DLLs list.

Mitigations

Note: The mitigations presented in this section are only applicable on Windows Vista SP1+ and require the critical Windows security update specified in KB2533623 on pre-Win8 machines. This security update was released in 2011 and it is quite reasonable to expect a machine patched to this level.

Explicit System DLL Loads

In all places where your code explicitly loads a system DLL via LoadLibrary or LoadLibraryEx, ensure that you use LoadLibraryEx and pass the LOAD_LIBRARY_SEARCH_SYSTEM32 flag. This ensures that only the system directory is searched for that specific DLL.

Note: In our testing, we found that passing the LOAD_LIBRARY_SEARCH_SYSTEM32 flag on unpatched Vista / Win7 systems can cause LoadLibraryEx to fail. If you need to support pre-KB2533623 versions of Vista/Win7, you will have to do a little more work. One good option is to look for the existence of the SetDefaultDllDirectories function, which was released in the same KB. If the function exists, it is safe to pass the LOAD_LIBRARY_SEARCH_SYSTEM32 flag.

Implicit DLL Loads

Calls to functions like WinVerifyTrust end up loading additional system DLLs and you have no way to control the LoadLibraryEx flags in this case.

To prevent this you can call the SetDefaultDllDirectories function at the start of your executable and pass in the LOAD_LIBRARY_SEARCH_SYSTEM32 flag. After this, the loader will only search the system directory unless lpFileName specifies a full path and/or override flags are passed into LoadLibraryEx.

After making this change, any loads of non-system DLLs must have the full path or non-zero LoadLibraryEx flags. Passing in a full path is a good practice in any case, as is doing a trust check on the DLL before loading it.

Note: Since SetDefaultDllDirectories is not available on unpatched Vista/Win7 systems it is a really good idea to dynamically load it from kernel32.dll.

Compile-Time DLL Imports

Some DLLs linked at compile-time (like kernel32.dll) will be loaded from the Known DLLs list.

However, there are many DLLs that are not on the Known DLLs list. This includes DLLs like winhttp.dll, sfc.dll, version.dll, lz32.dll and more.

In this case, you have two options:

  1. Switch to manually loading these DLLs and use GetProcAddress to get the imports.
  2. Mark the library for delay loading at link-time. This defers the DLL load to the first time one of its exports is called. Before the first call into the DLL, ensure that your executable has already called SetDefaultDllDirectories with the LOAD_LIBRARY_SEARCH_SYSTEM32 flag.The linker option for this is /DELAYLOAD:<dllname>.dll. You can find various pages on MSDN with this information, including this one.

    Note: You still link with the <dllname>.lib file and you also need to link with delayimp.lib as it handles the work of loading the DLL at runtime. Ex: for winhttp.dll, the linker options would be:
    winhttp.lib delayload.imp /DELAYLOAD:winhttp.dll

Bonus – Using LOAD_WITH_ALTERED_SEARCH_PATH

If someone forces you at gunpoint to support pre-Win8 systems without KB2533623 installed, you can get most of the way there. Keep in mind that there is no way to handle the case where a function like WinVerifyTrust dynamically loads additional DLLs as part of function execution.

  1. Always pass in the full path to LoadLibraryEx. For system DLLs, you can get the system directory using the GetSystemDirectory function.
  2. Always pass in the LOAD_WITH_ALTERED_SEARCH_PATH flag to LoadLibraryEx. This flag was introduced in Windows XP and helps with the following case:
    1. Dynamically load c:\windows\system32\foo.dll using LoadLibraryEx.
    2. Foo.dll is linked with bar.dll at compile time, and bar is also a system DLL.
      Without the LOAD_WITH_ALTERED_SEARCH_PATH, the search for bar.dll still begins in the executable directory. With the LOAD_WITH_ALTERED_SEARCH_PATH flag, the search path begins with the path to foo.dll, i.e. the system directory.
  3. Load DLLs manually at run-time wherever possible, and use GetProcAddress to get pointers to the required functions. DLLs like kernel32.dll and ntdll.dll will still be loaded when the executable starts, but they are on the Known DLLs list so that is not a problem.
Posted in Security | Tagged , | Leave a comment