We have mentioned DEP in several recent blog posts (1, 2, 3, and 4). This blog post will answer:
- What is DEP?
- How can you enable DEP?
- What are the risks in enabling different modes of DEP?
This is the first of a two-part blog series on DEP as a mitigation technology.
What is DEP?
DEP or “Data Execution Prevention” is a hardware + software solution for preventing the execution of code from pages of memory that are not explicitly marked as executable. Windows XP SP2, Windows Server 2003 SP1, and later operating systems check if the CPU supports enforcement of the ‘no execute’ or ‘execute disable bit’ for a page of memory.
Before Windows XP SP2, an exploit (“code”) could execute from memory allocated without the execute memory protection constantset. For example, if a page of memory were allocated using the VirtualAlloc() function specifying PAGE_READWRITE, it was still possible to execute code from that page of memory. Starting with Windows XP SP2 and Windows Server 2003 SP1, if the CPU supports the execute disable (XD – Intel CPUs) or no execute (NX – AMD CPUs) bits, any attempts to execute code from a page of memory with a (for example) PAGE_READWRITE memory protection will generate a STATUS_ACCESS_VIOLATION (0xC0000005) access violation exception. For more information on how hardware enforced DEP works please refer to the following article: http://technet.microsoft.com/en-us/library/bb457155.aspx.
DEP is “always on” for 64bit processes on 64bit versions of Windows and it cannot be disabled. The Windows DEP policy can be managed on both a system wide and per-process basis and both approaches will be discussed below. This blog post will apply specifically to 32bit processes running on either a 32bit or 64bit version of Windows.
How robust is DEP?
DEP presents a hurdle to attackers as they attempt to successfully exploit security vulnerabilities. In some cases, it is possible for an attacker to evade DEP by using an exploitation technique such as return-to-libc. DEP by itself is generally not a robust mitigation. DEP is a critical part of the broader set of exploit mitigation technologies that have been developed by Microsoft such as ASLR, SeHOP, SafeSEH, and /GS. These mitigation technologies complement one another; for example DEP’s weaknesses tend to be offset by ASLR and vice versa. DEP and ASLR used together are very difficult to bypass. The known bypasses that exist have been tied to specific application contexts (such as the IE7 and earlier bypass from Mark Dowd and Alex Sotirov).
How do I enable DEP?
If you have hardware that supports DEP and you are running Windows XP SP2 or newer operating system – you are already using DEP to some extent! Hardware enforced DEP can be configured system wide (applying to all processes) or using per-process policies. There are four different ways of enforcing a system wide DEP policy on platforms that support it.
- “Opt-In” – In this mode of operation DEP is enabled only for processes that explicitly opt-in to DEP. This is the default configuration for client operating systems such as Windows XP and Windows Vista.
- “Opt-Out” – In this mode of operation DEP is enabled by default for all processes except those that explicitly opt-out of DEP. This is the default configuration for server operating systems such as Windows Server 2003 and Windows Server 2008.
- “Always On” – In this mode of operation DEP is always enabled for all processes regardless of whether the program is compatible with DEP or not.
- “Always Off” – In this mode of operation DEP is always disabled for all processes.
Windows XP SP2 and Windows Server 2003 SP1
This article does a great job explaining how to use the boot.ini or the GUI to configure the memory protection settings on Windows XP SP2 and Windows Server 2003 SP1. Note that configuring the “system wide” DEP policy requires administrative privileges.
Windows Vista and Windows Server 2008
To configure the system-wide DEP policy on Windows Vista and Server 2008, modify the Boot Configuration Database using the bcdedit.exe console application from an elevated command prompt. You can find more information about how to do that here: http://msdn.microsoft.com/en-us/library/aa468629.aspx
Potential application compatibility issues with DEP “Always On”
Setting DEP to always be enabled for all processes may increase the risk of causing application problems due to certain behavior from versions of ATL prior to version 7.1. Older versions of ATL generated machine code at runtime and then attempted to execute this code from pages of memory that were not marked as executable, thus causing a DEP violation if DEP is enabled. If the system-wide setting is “opt-in”, a process known as “ATL thunk emulation” is used to avoid ATL related DEP crashes. “ATL thunk emulation” is not used if DEP is “Always On”. NOTE: When the “Always On” setting is used there are known compatibility issues with Microsoft Office applications when opening documents that contain VBA (Visual Basic for Applications) macros that can lead to DEP related crashes.
How can I tell if a specific process has opted-in or will opt-in to DEP?
The easiest way to tell if a process is currently using DEP is to use Process Explorer and to configure it to show the processes DEP status (View->Select Columns->Process DEP status) as shown below:
To determine why a process uses (or does not use) DEP requires a bit more investigative work. Here are the ways the system could have decided on a process’s DEP state. Each will be explored in more detail later in the blog post.
- The system-wide DEP policy was set to “AlwaysOn”. DEP is enabled for all processes.
- The system-wide DEP policy was set to “OptIn” and the process opted-in via an entry in the Application Compatibility Database.
- The system-wide DEP policy was set to “OptOut” and the process did not explicitly opt-out.
- The process executable was linked with the /NXCOMPAT linker switch and is running on a Vista or newer operating system.
- The process has the “ExecuteOptions” registry value set to 0 in the “Image File Execution Options” registry key and is running on a Windows Vista or newer operating system.
- The process called the new SetProcessDEPPolicy API on Vista SP1, XPSP3 or Windows Server 2008.
How Per-Process DEP policy works on Windows XP SP2 & Windows Server 2003 SP1
Windows XP SP2 and Windows Server 2003 SP1 included a heavy emphasis on security. As such, many of the default Windows programs were opted-in to DEP via the Application Compatibility database. To explore which programs opt-in to DEP on Windows XP SP2 or Windows Server 2003 SP1 you can download and install the latest Application Compatibility Toolkit and browse the main system database under the ‘Windows Components’ subset of applications.
All Windows XP SP2 applications opt-in to DEP if in category “Windows Components” with “AddProcessParametersFlags" compatibility fix applied:
The Windows Server 2003 SP1 default system wide policy is “OptOut”. That means all processes that do not explicitly opt-out will have DEP enabled. If the system-wide policy is changed, an application could still have DEP enabled by using the EnableDEP compatibility fix as shown below:
These compatibility fixes set the PEB’s ProcessParameters value to 0x00020000, causing Windows XP SP2 and Windows Server 2003 SP1 to opt a process in to DEP. We found 305 processes opted-in to DEP on Windows XP SP2 and Windows Server 2003 SP1 via the main application compatibility database (%windir%\AppPatch\sysmain.sdb) using this particular compatibility fix. We have attached the list of processes at the bottom of the blog.
New Per-Process DEP options on Windows Vista, Windows Server 2008 and newer operating systems
Microsoft has introduced options on newer operating systems to opt a process in to DEP.
/NXCOMPAT linker switch
On Vista and newer operating systems, processes linked with the /NXCOMPAT option are opted-in to DEP by default. Developers of 3rd party applications that want to take advantage of this mitigation technology can re-link their applications using the Visual C++ 2003 or newer version of the linker by passing in the “/NXCOMPAT” flag. This will cause the linker to emit a binary that declares that it supports DEP via an optional image header field supported on Vista and later operating systems. You can determine whether a process has been linked with the /NXCOMPAT linker switch using dumpbin.exe (which ships with Visual Studio):
C:\Windows\System32>dumpbin /headers dllhost.exe<br>Microsoft (R) COFF/PE Dumper Version 9.00.21022.08<br>Copyright (C) Microsoft Corporation. All rights reserved.<br><br><br>Dump of file dllhost.exe<br><br>PE signature found<br><br>File Type: EXECUTABLE IMAGE<br><br>FILE HEADER VALUES<br> 8664 machine (x64)<br> 5 number of sections<br> 4549BBFF time date stamp Thu Nov 02 05:35:59 2006<br> 0 file pointer to symbol table<br> 0 number of symbols<br> F0 size of optional header<br> 22 characteristics<br> Executable<br> Application can handle large (>2GB) addresses<br><br>OPTIONAL HEADER VALUES<br> 20B magic # (PE32+)<br> 8.00 linker version<br> 1400 size of code<br> 1000 size of initialized data<br> 0 size of uninitialized data<br> 1818 entry point (0000000100001818)<br> 1000 base of code<br> 100000000 image base (0000000100000000 to 0000000100006FFF)<br> 1000 section alignment<br> 200 file alignment<br> 6.00 operating system version<br> 6.00 image version<br> 6.00 subsystem version<br> 0 Win32 version<br> 7000 size of image<br> 400 size of headers<br> E1C5 checksum<br> 2 subsystem (Windows GUI)<br> 8140 DLL characteristics<br> Dynamic base<br> <b> NX compatible</b><br> Terminal Server Aware
ExecuteOptions registry setting
Windows Vista and Windows Server 2008 can also force a process to use DEP using the ExecuteOptions registry setting. ExecuteOptions can force a process to use DEP without changing the system-wide policy. Processes opted-in to DEP using the ExecuteOptions registry setting will be DEP-enabled for the life of the process (DEP cannot be turned off via any API calls). Compared to the system-wide Opt-In / Opt-Out approach, this could be seen as a disadvantage as discussed above regarding “ATL thunk emulation” when it comes to application compatibility. “ATL thunk emulation” is not used when a process is forced to use DEP via either the ExecuteOptions registry setting or /NXCOMPAT linker switch. Finally, the Windows Compatibility Database contains a list of DLLs that are known to be incompatible with DEP and when this registry setting is used, since DEP cannot be disabled for the process by the compatibility infrastructure in Windows, DEP related crashes may result if applications attempt to load DLLs that are not compatible with DEP.
Microsoft encourages 3rd party ISVs to support DEP (and other exploit mitigation technologies). The latest versions of Acrobat, Flash, QuickTime, and Sun’s Java runtime (JRE 6 Update 12) all support DEP. For more information on how to avoid ATL related DEP crashes when DEP is enabled for a process, refer to http://support.microsoft.com/kb/948468.
SetProcessDEPPolicy API
Finally, a new way of opting-in to DEP is available on our latest versions of Windows. Windows Vista SP1, Windows Server 2008 and Windows XP SP3 includ the SetProcessDEPPolicy API to allow an application to opt-in to DEP programmatically without /NXCOMPAT or being marked up in the appcompat database or registry. Exposing an API not only makes it easier for developers to opt-in to DEP but when using this API, ATL thunk emulation is performed and processes using older versions of ATL should not crash due to DEP violations. For example, Internet Explorer 8 takes advantage of this new functionality.
In part two of this series, we’ll demonstrate how to change an application’s DEP policy to mitigate a real-world attack.
- Robert Hensing, MSRC Engineering
*Postings are provided “AS IS” with no warranties, and confers no rights.*