On January 3rd, 2018, Microsoft released an advisory and security updates related to a newly discovered class of hardware vulnerabilities involving speculative execution side channels (known as Spectre and Meltdown) that affect AMD, ARM, and Intel CPUs to varying degrees. If you haven’t had a chance to learn about these issues, we recommend watching The Case of Spectre and Meltdown by the team at TU Graz from BlueHat Israel, reading the blog post by Jann Horn (@tehjh) of Google Project Zero, or reading the FOSDEM 2018 presentation by Jon Masters of Red Hat.
This new hardware vulnerability class represents a major advancement in CPU side channel attacks and we’re sincerely grateful to the researchers who discovered these issues, our industry partners, and the many individuals across Microsoft who have worked on these vulnerabilities. Mitigating hardware vulnerabilities through changes in firmware and software is a significant industry challenge, and there has been some confusion over the past two months for users as the industry has collectively continued to work on this.
In this post, we’ll provide insight into how Microsoft has approached mitigating speculative execution side channel hardware vulnerabilities to date. We’ll do this by briefly describing how we investigated this new vulnerability class, the taxonomy we established for reasoning about it, and the mitigations we have implemented as a result. This post is primarily geared toward security researchers and engineers who are interested in deeper technical details related to speculative execution side channel vulnerabilities and their respective mitigations. If you are interested in more general guidance, please refer to our knowledge base articles for Windows Server, Windows Client, and Microsoft cloud services.
Please note that the information in this post is current as of the date of this post.
Approaching the challenge
We first learned about speculative execution side channel vulnerabilities through the discoveries made by Jann Horn of Google Project Zero. Given the potential severity and the scope of these attacks, we kicked off our incident response process to drive coordination and remediation across Microsoft and with industry partners. This ultimately led to the mobilization of hundreds of individuals across the company, but before we could get there, we needed to assess the severity, impact, and root cause of the issue.
Conventional software vulnerabilities are well-understood and are relatively easy to perform root cause analysis on (we even have automation for many cases, see VulnScan). Speculative execution side channels, on the other hand, represented a fundamentally new hardware vulnerability class with no established process for determining their severity and their impact on existing software security models. To create this process, we and others in the industry needed to thoroughly research speculative execution side channels and establish a taxonomy and framework for reasoning about their effects and possible mitigations. The Microsoft Security Response Center (MSRC) brought in experts from across Microsoft (e.g. Microsoft Research Cambridge and the Windows Offensive Security Research team), and we hired Anders Fogh (@anders_fogh), of GDATA Advanced Analytics, as a consultant whose deep expertise on CPU side channel attacks greatly contributed to our understanding of these issues. This research gave us a foundation on which we and others were able to build.
A framework for speculative execution side channel vulnerabilities
Side channel attacks consist of three phases: the priming phase which is used to place a system into a desired initial state (e.g. flushing cache lines), the triggering phase which is used to perform the action that conveys information through the side channel, and the observing phase which is used to detect the presence of the information conveyed through the side channel. These phases can occur architecturally (by actually executing instructions), or speculatively (through speculative execution).
Side channel attacks involving speculative execution have four components: a speculation primitive , which provides the means for entering speculative execution down a non-architectural path; a windowing gadget , which provides a sufficient amount of time for speculative execution to convey information through a side channel; a disclosure gadget , which provides the means for communicating information through a side channel during speculative execution; and a disclosure primitive , which provides the means for reading the information that was communicated by the disclosure gadget. These four components can be combined in different ways to create a speculation technique.
Speculation primitives
Spectre and Meltdown make use of three different speculation primitives. The first is conditional branch misprediction (e.g. CVE-2017-5753, aka Variant 1), where speculation may occur when predicting whether a conditional branch is taken or not taken. The second is indirect branch misprediction (e.g. CVE-2017-5715, aka Variant 2), where speculation may occur when predicting the target of an indirect branch (including call, jump, and return). The third deals with exception delivery or deferral (e.g. CVE-2017-5754, aka Variant 3), where speculative execution may continue past the point at which a fault will be raised. Each of these primitives can be used to enter speculative execution along a non-architectural path and attackers can attempt to trigger speculation in different ways (e.g. through training or colliding of predictors).
Windowing gadgets
Speculative execution is effectively a race condition between the retirement of the µop (micro-operation) that led to speculation and the µop being executed speculatively. Attackers must win this race condition for a disclosure gadget to execute speculatively. We have defined three categories of windowing gadgets. The first deals with non-cached loads that trigger a load from main memory which typically takes hundreds of cycles on most CPUs. The second involves a dependency chain of loads of which may or may not be non-cached. The third involves a dependency chain of integer ALU operations which have fixed-latency operations. These windowing gadgets can occur naturally in code, or they can be manufactured by attackers in certain cases.
Disclosure gadgets
The primary purpose of speculative execution in the context of a side channel attack is to trigger the execution of a disclosure gadget. This is because the disclosure gadget is responsible for accessing information that is stored in a security domain which is not architecturally visible to the attacker and conveying that information through a side channel. This could include memory content, register state, and so on. As with gadgets used in Return Oriented Programming (ROP), there are many possible examples of disclosure gadgets and their form and function will vary.
Disclosure primitives
After speculative execution of a disclosure gadget has conveyed information via a side channel, the final step is to detect the presence of that information using a disclosure primitive. Each disclosure primitive is intrinsically linked to the communication channel through which information is conveyed (e.g., CPU data cache, etc.), and the manner through which a disclosure gadget conveyed the information. There has been a significant amount of prior research into disclosure primitives for other side channels, such as FLUSH+RELOAD and PRIME+PROBE. These primitives are generally applicable to speculative execution side channels as well.
Relevance to software security models
Modern operating systems and applications depend on software security models that isolate native code on a device. These security models include concepts like virtualization-based isolation, kernel/user separation, process-based isolation, and language-based isolation (e.g. JavaScript runtime sandbox). The designs for these security models take a strong dependency on the underlying hardware guarantees that memory allocated to one security domain cannot be read from or written to by another security domain unless permitted by a privileged arbiter (such as a hypervisor, the kernel, or a managed runtime). This is enforced by the hardware and operating system through concepts like privilege modes (hypervisor, kernel, user), paging, and so on. Unfortunately, speculative execution side channel vulnerabilities can make it possible to read memory across security domains, thus violating existing software security models.
To illustrate this, the following table summarizes the relevance of speculation primitives to the attack scenarios that represent software security models of concern. Each attack scenario is described in terms of the direction that information flows when performing a speculative execution side channel attack. Orange cells indicate that the speculation primitive is applicable to the corresponding attack scenario whereas grey cells are not applicable.
| Attack Category | Attack Scenario | Conditional branch misprediction | Indirect branch misprediction | Exception delivery or deferral |
|---|---|---|---|---|
| Inter-VM | Hypervisor-to-guest | |||
| Host-to-guest | ||||
| Guest-to-guest | ||||
| Intra-OS | Kernel-to-user | |||
| Process-to-process | ||||
| Intra-process | ||||
| Enclave | Enclave-to-any |
Inter-VM
This category deals with attacks on virtualization-based isolation which relies on hardware support for virtualization extensions for security. Hypervisor-to-guest attacks involve a malicious guest attempting to read hypervisor memory. Host-to-guest attacks involve a malicious guest reading the memory of the privileged guest that provides virtualization assists (e.g. the root partition in Hyper-V or dom0 in Xen). Guest-to-guest attacks involve a malicious guest reading the memory of another guest.
Intra-OS
This category deals with attacks within a traditional instance of an operating system (e.g. the OS in a VM or on a physical device). Kernel-to-user attacks involve a malicious user mode process reading kernel memory. Process-to-process attacks involve a malicious user mode process reading the memory of another process that should not be readable. Intra-process attacks involve a process being coerced into reading its own memory such as the case of a web site providing JavaScript to a web browser which then creates the conditions for a speculative execution side channel vulnerability.
Enclave
The enclave category deals with enclave-to-any attacks where code executing outside of an enclave (e.g. an Intel SGX enclave) is able to read memory within the enclave.
Mitigations for speculative execution side channel vulnerabilities
The taxonomy and framework described above provide the basis for defining a strategy and a set of tactics for mitigating speculative execution side channel vulnerabilities. Through this process we identified three general tactics that can be employed by software (and hardware) to mitigate this issue with varying levels of completeness. These tactics are summarized in the following table:
| Prevent speculation techniques | Remove sensitive content from memory | Remove observation channels |
|---|---|---|
| SSpeculative execution attacks inherently rely on using a speculation primitive to execute a desired disclosure gadget. These attacks can be mitigated by preventing the use of a speculation primitive or specific instance of a speculation technique. This tactic is desirable because it mitigates the issue at or near the root cause. | Speculative execution attacks rely on sensitive information being accessible in the victim’s address space. These attacks can be mitigated by removing sensitive information from memory such that it is not readable during speculative execution. This approach cannot protect against reading of register state or “non-sensitive” memory content (such as address space information). | Speculative execution attacks inherently rely on the ability to communicate information through a side channel (such as CPU data caches). These attacks can be mitigated by removing channels for communication which thereby prevents the use of certain disclosure primitives. This can provide a broad mitigation that is independent of any single speculation primitive. |
In the sections that follow, we will describe the mitigations we have implemented and what impact they have to the speculation techniques that have been described thus far.
Preventing speculation techniques
One of the best ways to mitigate vulnerabilities is by eliminating a class of issues at the root cause. For speculative execution attacks, this can be accomplished by preventing speculation primitives from being used to execute a desired disclosure gadget.
Speculation barrier via execution serializing instruction
This mitigation involves inserting instructions that act as a speculation barrier by serializing execution. On AMD and Intel CPUs this involves the use of an LFENCE instruction whereas ARM recommends the use of conditional select/move (CSEL/MOVS) instructions along with a new explicit barrier instruction (CSDB). Microsoft has added support for the /Qspectre flag to Visual C++ which currently enables some narrow compile-time static analysis to identify at-risk code sequences related to CVE-2017-5753 and insert speculation barrier instructions. This flag has been used to rebuild at-risk code in Windows and was released with our January 2018 security updates. It is important to note, however, that the Visual C++ compiler cannot guarantee complete coverage for CVE-2017-5753 which means instances of this vulnerability may still exist. We recommend that software developers treat CVE-2017-5753 as a new hardware vulnerability class that can be mitigated in software by adding an explicit speculation barrier. While we are continuing to look for opportunities to improve Visual C++ support for /Qspectre, Microsoft has also announced a Speculative Execution Side Channel bounty program to encourage researchers to find and report any instances of CVE-2017-5753 that may remain.
In a similar vein, the JavaScript Just-in-Time (JIT) compiler in Microsoft Edge and Internet Explorer includes code generation mitigations for CVE-2017-5753. These mitigations constrain the permissible behavior of speculative execution related to CVE-2017-5753 for dynamically generated native code that is produced by the JIT.
Security domain CPU core isolation
Modern CPUs typically store prediction state in per-core or per-SMT caches. This means that isolating workloads to distinct cores can be used to robustly mitigate speculation techniques that rely on colliding prediction state. To that end, Microsoft has published documentation describing how minimum root (“minroot”) can be used to dedicate core(s) to the Hyper-V root partition and how CPU groups can be used to dedicate core(s) to guests.
Indirect branch speculation barrier on demand and mode change
Speculation techniques that rely on one security domain (user, kernel, hypervisor) being able to influence indirect branch predictions (e.g. CVE-2017-5715) in another security domain can be mitigated through the use of additional hardware interfaces provided by Intel, AMD, and ARM. In the case of Intel and AMD, these hardware interfaces require microcode updates to expose the features of which operating systems can take advantage. These features provide software with the ability to flush indirect branch prediction state, inhibit the use of branch predictions from less-privileged security domains, and protect against sibling hardware thread prediction interference. The microcode updates from Intel and AMD are currently at varying levels of readiness and availability, but they are generally expected to enable strong mitigations for speculation techniques that use indirect branch mispredictions across security domains. On March 1st, 2018, Microsoft announced the availability Intel microcode updates through the Microsoft Update Catalog (KB4090007).
Non-speculated or safely-speculated indirect branches
Some types of indirect branches on existing Intel and AMD hardware are either not predicted at all or have generally safe prediction behavior and can therefore be used to mitigate CVE-2017-5715. For example, far JMP on existing Intel hardware is not predicted and can therefore be used as an alternative to near indirect CALL and JMP. This mitigation takes advantage of the far JMP behavior by transforming all at-risk indirect CALL and JMP instructions into far JMP for the Hyper-V hypervisor when running on Intel hardware. For the Hyper-V hypervisor running on AMD hardware, at-risk near indirect CALL and JMP are preceded by an execution serializing RDTSCP instruction as per AMD’s recommended guidance.
In a similar vein, Google has also proposed a software solution known as retpoline which transforms near indirect calls and jumps into “retpolines” that rely on safe and deterministic near return mispredictions when transferring control. This solution provides a strong mitigation for speculation techniques that use indirect branch misprediction on many existing CPUs. We believe retpoline can be a suitable mitigation for environments where it is practical to rebuild all at-risk code and we have pursued a similar path through the changes we have made to Hyper-V. We are also evaluating performance optimizations involving a hybrid mitigation for the Windows kernel and device drivers that would involve retpoline and hardware support for indirect branch speculation barriers.
There are some additional points that are important to consider regarding the use of retpoline. Some modern CPUs, as noted by Intel, do not satisfy the security properties that retpoline relies upon for near-return prediction without requiring software to prevent RSB (Return Stack Buffer) underflows. These CPUs require software to perform RSB stuffing to prevent underflow which we believe to be nontrivial for software to perform in the general case. In addition, scenarios that would require all at-risk software to be rebuilt face significant challenges in the context of expansive, multi-vendor, binary ecosystems such as multi-tenant cloud environments or traditional desktops and servers that make use of applications and other software produced by multiple vendors. For these environments, we believe it is unreasonable to expect that all at-risk software will be rebuilt. As we look toward the future, hardware security improvements such as Intel’s Control-flow Enforcement Technology (CET) will encounter compatibility issues with the use of retpoline which software developers need to consider.
Removing sensitive content from memory
All attacks involving speculative execution side channels attempt to gain access to information across security domains. This means that attackers will attempt to disclose data from a victim security domain that they should not have access to. As such, removing sensitive information from the victim security domain can be an effective method of preventing information disclosure using speculation techniques.
Hypervisor address space segregation
The Microsoft Hyper-V hypervisor has historically maintained an identity map of all physical memory to accelerate memory accesses while executing in the hypervisor (known as the “physical map”). To minimize exposure to speculation techniques, we have removed the physical map entirely and no longer map all physical memory into the address space of the hypervisor, thus helping to mitigate the risk of cross-VM information disclosure through speculative execution.
Split user and kernel page tables
This mitigation is known as Kernel Virtual Address (KVA) Shadow on Windows and it mitigates the speculation technique known as Meltdown (CVE-2017-5754) by creating two page directory bases per process. The first maps the “user” page tables which only contains user mode mappings and a small number of kernel transition pages. The second maps the “kernel” page tables which contains both user and kernel mappings for the process. The user page tables are active while executing code in user mode while the kernel pages are switched to when trapping and executing code in kernel mode on behalf of a process. This has the effect of removing sensitive kernel memory content from the virtual address space for a process which thereby provides a robust mitigation for CVE-2017-5754. This mitigation draws inspiration from prior research known as KAISER, although KVA Shadow is not intended to enable robust local Kernel ASLR (KASLR). KVA Shadow is also similar to mitigations available for the Linux kernel (Kernel Page Table Isolation, KPTI), Apple MacOS and iOS, and Google Chromebook.
Removing observation channels
All speculation techniques implicitly rely on communicating information through a side channel in a manner that can be detected by a disclosure primitive. This means speculation techniques can be broadly mitigated by removing channels for communicating and observing information.
Map guest memory as noncacheable in the root extended page tables
Virtualization software often maps guest memory into the address space of a privileged guest (the “root” for Hyper-V, or dom0 for Xen) to enable fast communication through shared memory. In the context of speculation techniques, these shared memory regions can be used to communicate information through the loading of a shared cache line (e.g. FLUSH+RELOAD). One way to mitigate this is by mapping guest memory regions as noncacheable (UC) in the extended page tables for the root. On all CPUs, noncacheable memory cannot be loaded during speculative execution and therefore prevents loading of a shared cache line.
Do not share physical pages across guests
In some cases, virtualization software may share physical pages between guests. These shared memory regions can be used to communicate information through the loading of a shared cache line (similar to mapping guest memory as noncacheable). In this case, these shared regions can facilitate guest-to-guest attacks involving speculative execution. This mitigation implements the straightforward solution which is to stop sharing physical pages between guests that are not part of the same security domain.
Decrease browser timer precision
The intra-process attack scenario is relevant to web browsing scenarios where JavaScript provided by a website may induce the JavaScript JIT to create native code that manifests a speculative execution side channel vulnerability such as CVE-2017-5753. To help mitigate this, Microsoft Edge and Internet Explorer have been updated to reduce the precision of timers exposed by the web browser. This increases the difficulty of disclosing information with high bandApplicability of mitigations
The complex nature of these issues makes it difficult to understand the relationship between mitigations, speculation techniques, and the attack scenarios to which they apply.
The legend for the tables that follow is:
| Applicable | Not applicable |
Mitigation relationship to attack scenarios
The following table summarizes the relationship between attack scenarios and applicable mitigations.
| Mitigation Tactic | Mitigation Name | Inter-VM | Intra-OS | Enclave |
|---|---|---|---|---|
| Prevent speculation techniques | Speculation barrier via execution serializing instruction | |||
| Security domain CPU core isolation | ||||
| Indirect branch speculation barrier on demand and mode change | ||||
| Non-speculated or safely-speculated indirect branches | ||||
| Remove sensitive content from memory | Hypervisor address space segregation | |||
| Split user and kernel page tables (“KVA Shadow”) | ||||
| Remove observation channels | Map guest memory as noncacheable in root extended page tables | |||
| Do not share physical pages across guests | ||||
| Decrease browser timer precision |
Mitigation relationship to variants
The following table summarizes the relationship between Spectre and Meltdown variants and applicable mitigations.
| Mitigation Tactic | Mitigation Name | CVE-2017-5753 (variant 1) | CVE-2017-5715 (variant 2) | CVE-2017-5754 (variant 3) |
|---|---|---|---|---|
| Prevent speculation techniques | Speculation barrier via execution serializing instruction | |||
| Security domain CPU core isolation | ||||
| Indirect branch speculation barrier on demand and mode change | ||||
| Non-speculated or safely-speculated indirect branches | ||||
| Remove sensitive content from memory | Hypervisor address space segregation | |||
| Split user and kernel page tables (“KVA Shadow”) | ||||
| Remove observation channels | Map guest memory as noncacheable in root extended page tables | |||
| Do not share physical pages across guests | ||||
| Decrease browser timer precision |
Wrapping up
In this post, we described our approach toward mitigating a new hardware vulnerability class involving speculative execution side channels. We believe the mitigations described in this post offer strong protections for these vulnerabilities. Going forward, we recommend that the software industry view these issues as a new vulnerability class that may require software changes in order to help mitigate (e.g. like buffer overruns, type confusions, use after frees, and so on). We expect this new hardware vulnerability class to be the subject of further research as we’ve witnessed in the past with other vulnerability classes. Microsoft is committed to protecting our customers, and we will continue to work with our industry partners on mitigating speculative execution side channel vulnerabilities. To reinforce this commitment, we have launched the Speculative Execution Side Channel bounty program to encourage discovery and reporting of these vulnerabilities such that they can be fixed.
Matt Miller, Microsoft Security Response Center (MSRC)