MEX is Your Friend: Analyzing 32-bit Processes in a 64-bit Kernel Dump

Introduction

In 2016, Microsoft publicly released a WinDBG extension called MEX. It has a number of commands that make life easier when kernel debugging or analyzing a crash dumps. You can download it here if you want to give it a try.

The Problem

I was recently asked to look a hang issue in Microsoft Outlook. I was given a full memory dump taken on a 64-bit system, which seemed like overkill (it wasn’t). It’s generally quite straightforward to analyze user mode apps from a memory dump — see the following OSR article for more info: Analyst’s Perspective: Analyzing User Mode State from a Kernel Connection.

I started my debugging session and quickly ran into a snag: Outlook was running as 32-bit process, leading to stacks with mostly wow64* calls in them.

0: kd> !process 0 0 outlook.exe
PROCESS fffffa800b6e6060
 SessionId: 1 Cid: 1408 Peb: 7efdf000 ParentCid: 1268
 DirBase: a9baa000 ObjectTable: fffff8a0083de430 HandleCount: 6880.
 Image: OUTLOOK.EXE

0: kd> .process /r /p fffffa800b6e6060
Implicit process is now fffffa80`0b6e6060
Loading User Symbols
.....
0: kd> !process fffffa800b6e6060
PROCESS fffffa800b6e6060
 SessionId: 1 Cid: 1408 Peb: 7efdf000 ParentCid: 1268
 DirBase: a9baa000 ObjectTable: fffff8a0083de430 HandleCount: 6880.
 Image: OUTLOOK.EXE
[...snip...]
 THREAD fffffa800b858060 Cid 1408.180c Teb: 000000007efaa000 Win32Thread: fffff900c1d3f010 WAIT: (UserRequest) UserMode Non-Alertable
[...snip...]
 fffff880`093dbec0 fffff800`03484e42 nt!KiSwapContext+0x7a
 fffff880`093dc000 fffff800`034918da nt!KiCommitThreadWait+0x1d2
 fffff880`093dc090 fffff800`0378b1af nt!KeWaitForMultipleObjects+0x272
 fffff880`093dc350 fffff800`037b8fc9 nt!ObpWaitForMultipleObjects+0x294
 fffff880`093dc820 fffff800`0348e693 nt!NtWaitForMultipleObjects32+0xec
 fffff880`093dca70 00000000`74cc2e09 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`093dcae0)
 00000000`0352ebe8 00000000`74cc283e wow64cpu!CpupSyscallStub+0x9
 00000000`0352ebf0 00000000`74d3d286 wow64cpu!WaitForMultipleObjects32+0x3b
 00000000`0352ecb0 00000000`74d3c69e wow64!RunCpuSimulation+0xa
 00000000`0352ed00 00000000`773898ec wow64!Wow64LdrpInitialize+0x42a
 00000000`0352f250 00000000`7734a36e ntdll! ?? ::FNODOBFM::`string'+0x22b74
 00000000`0352f2c0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

The Painful Solution

After asking around, I found out that I could use .thread /w to look at 32-bit stacks but the method proved cumbersome, as I was looking for a particular thread out of 57 threads in the process. I was too lazy to write a script and so I spent over an hour going through each stack in the process like so:

0: kd> .thread /w fffffa800b858060 ; kb ; .effmach amd64
Implicit thread is now fffffa80`0b858060
x86 context set
 *** Stack trace for last set context - .thread/.cxr resets it
 # ChildEBP RetAddr Args to Child 
00 039afa64 755e171a 00000005 039afab4 00000001 ntdll_77500000!NtWaitForMultipleObjects+0x15
01 039afb00 76c019fc 039afab4 039afb28 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100
02 039afb48 76a90882 00000005 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0
03 039afb9c 6b16df8b 000003a0 039afbe4 ffffffff USER32!RealMsgWaitForMultipleObjectsEx+0x14d
WARNING: Stack unwind information not available. Following frames may be wrong.
04 039afbbc 2f7a5538 00000004 039afbe4 00000000 mso!MsoHrSetupHTMLImport+0x392
05 039afc10 62398646 00000000 0775e2d4 623985a6 OUTLOOK!HrMsgDownloadedNotification+0x3ff
06 039afc34 6b0a08e6 0775e2d4 00000000 572383a4 olmapi32!MSProviderInit+0x95f
07 039afc78 6b09e0fd 1191f900 039afd10 0039d16c mso!Ordinal3464+0x4a2
08 039afc98 6b09ddc3 039afd10 00000000 039afcf4 mso!Ordinal2771+0x450
09 039afcb4 6b09c027 039afd10 00000000 003997f0 mso!Ordinal2771+0x116
0a 039afce8 6b0962b7 003997f0 00000000 6b0962b7 mso!Ordinal2929+0x209
0b 039afd44 76c0336a 003997f0 039afd90 77539902 mso!Ordinal4724+0x67
0c 039afd50 77539902 003997f0 74fcff46 00000000 kernel32!BaseThreadInitThunk+0xe
0d 039afd90 775398d5 6b09625f 003997f0 ffffffff ntdll_77500000!__RtlUserThreadStart+0x70
0e 039afda8 00000000 6b09625f 003997f0 00000000 ntdll_77500000!_RtlUserThreadStart+0x1b
Effective machine: x64 (AMD64)

The MEX Solution

My solution worked but it was incredibly inefficient. The next morning I was still looking at the dump and decided to play around with MEX to see if it could help me out with something else. As it turned out, it was also able to help with my initial problem. Instead of spending an hour+ pasting commands into the debugger, I was able to get to the same point in about 1 minute.

0: kd> !mex.p fffffa800b6e6060
Name Address Ses PID Parent PEB Create Time Mods Handle Thrd User Name
=========== ======================== === ============= ============= ================ ========================== ==== ====== ==== ==========
OUTLOOK.EXE fffffa800b6e6060 (E|K|O) 1 1408 (0n5128) 1268 (0n4712) 000000007efdf000 04/26/2017 03:52:59.367 AM 281 6880 57 BOS\zy10os

Command Line: "C:\Program Files (x86)\Microsoft Office\Office14\OUTLOOK.EXE"

Memory Details:

VM Peak Work Set Commit Size PP Quota NPP Quota
 ========= ========= ========= =========== ======== =========
 963.62 MB 980.07 MB 281.02 MB 221.82 MB 1.36 MB 206 KB

Show LPC Port information for process

Show Threads: Unique Stacks !mex.listthreads (!lt) fffffa800b6e6060 !process fffffa800b6e6060 7

0: kd> !mex.lt fffffa800b6e6060
Process PID Thread Id State Time Reason Waiting On
=============== ==== ================ ==== ======= ============= ============== =================================================
OUTLOOK.EXE *32 1408 fffffa8007592060 fc0 Waiting 11s.138 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8007588840 1814 Waiting 9s.001 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b4dbb50 14bc Waiting 15ms UserRequest 
OUTLOOK.EXE *32 1408 fffffa80082c3b50 1808 Waiting 15ms UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b858060 180c Waiting 11s.138 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b76bb50 11bc Waiting 15ms WrQueue 
OUTLOOK.EXE *32 1408 fffffa800ad06b50 1910 Waiting 55s.551 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b552b20 1544 Waiting 11s.122 WrUserRequest 
OUTLOOK.EXE *32 1408 fffffa8007511060 15a4 Waiting 11s.122 WrUserRequest 
OUTLOOK.EXE *32 1408 fffffa8007404060 1670 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b71ab50 1930 Waiting 36m:05.449 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8007578060 1a5c Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8007581b50 1a0c Waiting 1m:28.733 WrQueue 
OUTLOOK.EXE *32 1408 fffffa8008fe9060 11b8 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b85b060 1590 Waiting 1s.747 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800a1a4060 cbc Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa80072f3b50 1820 Waiting 9s.984 WrLpcReply Thread: fffffa800c877060 in mcshield.exe (0n3120)
OUTLOOK.EXE *32 1408 fffffa80074a4b50 1228 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8007584b50 58c Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8009af1060 15c4 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800983e7c0 648 Waiting 6m:18.598 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b3b24b0 16b4 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b87f060 16d8 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800adc5060 1668 Waiting 1s.778 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b67e6c0 1788 Waiting 1m:39.949 UserRequest 
OUTLOOK.EXE *32 1408 fffffa80075a5b50 1658 Waiting 38s.017 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b52eb50 c20 Waiting 21m:28.505 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b40cb50 bd4 Waiting 15ms UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b375060 e00 Waiting 1m:31.993 WrUserRequest 
OUTLOOK.EXE *32 1408 fffffa800b2d67f0 10c0 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b6d8060 1990 Waiting 3m:59.196 WrUserRequest 
OUTLOOK.EXE *32 1408 fffffa800ba53b50 16e0 Waiting 6m:18.598 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800baa1060 1a98 Waiting 9m:41.197 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b697060 14a8 Waiting 15h:22:24.615 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800ba2e060 15ec Waiting 1m:30.511 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800bb68060 1924 Waiting 1m:30.511 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800ba74060 1130 Waiting 35s.271 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800bad2060 3e0 Waiting 1m:31.744 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800b700060 1900 Waiting 11s.122 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8007362640 18a0 Waiting 1m:29.528 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800bab5060 ce0 Waiting 1m:30.932 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8009e3f060 15fc Waiting 15h:22:24.615 WrQueue 
OUTLOOK.EXE *32 1408 fffffa800be1f640 1db8 Waiting 1m:31.416 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800689f060 1784 Waiting 6m:51.905 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8006b59710 1f88 Waiting 44s.304 WrUserRequest 
OUTLOOK.EXE *32 1408 fffffa800bd60900 21a0 Waiting 15m:41.044 UserRequest 
OUTLOOK.EXE *32 1408 fffffa8006c70060 1c5c Waiting 15ms UserRequest 
OUTLOOK.EXE *32 1408 fffffa800c253710 1850 Waiting 280ms UserRequest 
OUTLOOK.EXE *32 1408 fffffa800befbb50 2074 Waiting 55s.661 DelayExecution 
OUTLOOK.EXE *32 1408 fffffa8009e95840 2164 Waiting 9s.001 WrQueue 
OUTLOOK.EXE *32 1408 fffffa800c71a650 22b8 Waiting 4m:01.567 WrQueue 
OUTLOOK.EXE *32 1408 fffffa800c000870 2314 Waiting 15ms WrQueue 
OUTLOOK.EXE *32 1408 fffffa8009ede2b0 2014 Waiting 15ms WrQueue 
OUTLOOK.EXE *32 1408 fffffa800bef7460 1d24 Waiting 15ms WrQueue 
OUTLOOK.EXE *32 1408 fffffa800c28f060 1848 Waiting 11s.232 WrQueue 
OUTLOOK.EXE *32 1408 fffffa800c0a2b50 1258 Waiting 11s.122 UserRequest 
OUTLOOK.EXE *32 1408 fffffa800bfde060 21a8 Waiting 15ms WrQueue

Thread Count: 57

0: kd> !mex.t fffffa800b858060
Process Thread CID TEB UserTime KernelTime ContextSwitches Wait Reason Time State COM-Initialized
OUTLOOK.EXE *32 (fffffa800b6e6060) fffffa800b858060 (E|K|W|R|V) 1408.180c 000000007efaa000 312ms 94ms 73230 UserRequest 11s.138 Waiting APTKIND_MULTITHREADED (MTA)

[...snip...]

# Child-SP Return Call Site
0 fffff880093dbec0 fffff80003484e42 nt!KiSwapContext+0x7a
1 fffff880093dc000 fffff800034918da nt!KiCommitThreadWait+0x1d2
2 fffff880093dc090 fffff8000378b1af nt!KeWaitForMultipleObjects+0x272
3 fffff880093dc350 fffff800037b8fc9 nt!ObpWaitForMultipleObjects+0x294
4 fffff880093dc820 fffff8000348e693 nt!NtWaitForMultipleObjects32+0xec
5 fffff880093dca70 0000000074cc2e09 nt!KiSystemServiceCopyEnd+0x13
0 00000000039afa64 00000000755e171a ntdll_77500000!NtWaitForMultipleObjects+0x15
1 00000000039afa6c 0000000076c019fc KERNELBASE!WaitForMultipleObjectsEx+0x100
2 00000000039afb08 0000000076a90882 kernel32!WaitForMultipleObjectsExImplementation+0xe0
3 00000000039afb50 000000006b16df8b USER32!RealMsgWaitForMultipleObjectsEx+0x14d
4 00000000039afba4 000000002f7a5538 mso!MsoHrSetupHTMLImport+0x392
5 00000000039afbc4 0000000062398646 OUTLOOK!HrMsgDownloadedNotification+0x3ff
6 00000000039afc18 000000006b0a08e6 olmapi32!MSProviderInit+0x95f
7 00000000039afc3c 000000006b09e0fd mso!Ordinal3464+0x4a2
8 00000000039afc80 000000006b09ddc3 mso!Ordinal2771+0x450
9 00000000039afca0 000000006b09c027 mso!Ordinal2771+0x116
a 00000000039afcbc 000000006b0962b7 mso!Ordinal2929+0x209
b 00000000039afcf0 0000000076c0336a mso!Ordinal4724+0x67
c 00000000039afd4c 0000000077539902 kernel32!BaseThreadInitThunk+0xe
d 00000000039afd58 00000000775398d5 ntdll_77500000!__RtlUserThreadStart+0x70
e 00000000039afd98 0000000000000000 ntdll_77500000!_RtlUserThreadStart+0x1b

Switch to x86 - to switch back to x64/amd64 run !sw

Conclusion

MEX is your friend.

Advertisements
This entry was posted in Debugging, Windbg. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s