Summary
Microsoft recently mitigated a vulnerability in Azure Data Factory and Azure Synapse pipelines. The vulnerability was specific to the third-party Open Database Connectivity (ODBC) driver used to connect to Amazon Redshift in Azure Synapse pipelines and Azure Data Factory Integration Runtime (IR) and did not impact Azure Synapse as a whole. The vulnerability could have allowed an attacker to perform remote command execution across IR infrastructure not limited to a single tenant.
Microsoft has conducted a detailed internal investigation to identify any cases of abuse. The only activity identified was performed by Orca Security, who reported the vulnerability. Our investigation found no evidence of misuse or malicious activity. The vulnerability was mitigated on April 15, 2022.
There is no action needed from Azure Data Factory or Azure Synapse pipeline customers who are hosted in the Azure cloud (Azure Integration Runtime) or who host on-premises (Self-Hosted Integration Runtime) with auto-updates turned on. Self-host IR customers without auto-update need to take action to safeguard their deployments. Customers with this configuration were notified and provided guidance through Azure Service Health Alerts (Tracking ID: MLC3-LD0); however, additional information can be found below in the “Customer Recommendations and Additional Support” section. Customers with auto-updates enabled do not need to take additional action.
The following sections explain in more detail the relevant architectural background of the services and components involved, some high-level technical details of the vulnerability and steps Microsoft has taken to mitigate the issue and, any next steps or recommendations for customers.
Background
Azure Data Factory is a Microsoft Cloud Extract Transform Load (ETL) service that enables data integration and data transformation. Azure Data Factory is available as a standalone service, and it is also provided as Azure Synapse pipelines.
Customers using Azure Data Factory or Azure Synapse pipelines can create an Integration Runtime (IR) in their factories and/or workspaces to allow for data integration across different network environments. Azure Synapse pipelines can be used to integrate data from various sources into Synapse Analytics workspaces. These pipelines also support connectors, which allow data to be integrated across different data stores, including third-party products. Azure Data Factory and Azure Synapse pipelines have many data connectors to different data sources built-in today.
IRs using Azure Synapse pipelines can be hosted in the Azure cloud (via the Azure Data Factory Integration Runtime) or hosted on-premises (Self-Hosted Integration Runtime). Cloud-hosted Azure IRs can also be configured with a Managed Virtual Network (VNet) and will use private endpoints to connect to supported data stores, which can provide an extra layer of network isolation.
As a high-level architectural overview of the hosting models:
- Azure IR (with a Managed Virtual Network) : Cloud-hosted Azure IRs with a Managed VNet provide a dedicated container and dynamic pool behind a VNet, it is not shared across multiple customers.
- Azure IR (without a Managed Virtual Network) : Cloud-hosted Azure IRs execute pipeline activities in a shared pool of underlying compute resources. This allows multiple customers to utilize the resources in this pool and dynamically scale nodes at runtime.
- Self-hosted IR (SHIR) : As SHIR requires an on-premises or customer-provided virtual machine to execute tasks, SHIRs are dedicated to a single customer by design. SHIR can pull tasks from cloud or other on-premises data sources.
The vulnerability was specific to the third-party ODBC connector used to connect to Amazon Redshift in Azure Synapse pipelines and Azure Data Factory Integration Runtime (IR).
Vulnerability Impact
The vulnerability in the third-party ODBC connector for Amazon Redshift allowed a user running jobs in a Synapse pipeline to execute remote commands. A user who exploited this vulnerability could then potentially acquire the Azure Data Factory service certificate and execute commands in another tenant’s Azure Data Factory Integration Runtimes. These certificates are specific to Azure Data Factory and Synapse Pipelines, and do not pertain to the rest of Azure Synapse.
Investigation and Mitigation
Orca Security reported a vulnerability to Microsoft on January 4, 2022, at which point we began our internal investigation to identify the scope of impact and to protect customers. Our timeline for investigation and mitigation can be summarized as follows:
- January 4 - Orca reported the issue to Microsoft
- March 2 - Microsoft completed rollout of initial hotfix
- March 11 - Microsoft identified and notified customers affected by the researcher’s activity
- March 30 – Orca notified Microsoft of an additional attack path to the same vulnerability
- April 13 – Orca notified Microsoft of a second attack path to the same vulnerability
- April 15 – Additional fixes deployed for the two newly reported attack paths as well as additional defense in depth measures applied
Microsoft fully mitigated attack paths to this vulnerability on April 15, 2022, by taking the following steps across all IR types:
- Mitigated remote command execution in the impacted driver
- Reduced the job execution privilege in the Azure Integration Runtime
- Added extra validation layers as a defense in depth to harden the service
- Contained and closely monitored the backend certificate for adversary activity and pivots, before rotation and revocation
- Rotated and revoked the backend service certificate and other Microsoft credentials that were accessed by the finder
- Microsoft added additional defense in depth to backend API’s by moving to using activity isolated time-bound tokens instead of certificates
- Collaborated with the third-party ODBC driver provider on root-cause fixes to the driver used to connect to Amazon Redshift
- Reviewed third-party driver vendor code and ran our security tooling to ensure it meets our security standards
Detections
While our investigation found no evidence of Microsoft product or service misuse or malicious activity from this vulnerability aside from the activity Orca reported, we are sharing the following Microsoft Defender for Endpoint and Microsoft Defender Antivirus detections to protect customers.
-
Customers using automatic updates do not need to take additional action. Enterprise customers managing updates should select the detection build 1.363.1065.0 or later and deploy it across their environments.
-
Microsoft Defender Antivirus version 1.363.1065.0 or later detects components and behaviors related to this threat and protects customers through the following detections:
- Behavior:Win32/SuspAzureRequest.A
- Behavior:Win32/SuspAzureRequest.B
- Behavior:Win32/SuspAzureRequest.C
- Behavior:Win32/LaunchingSuspCMD.B
-
Microsoft Defender for Endpoint alerts with the following titles in the Microsoft 365 Defender portal can indicate threat activity on your network:
- Suspicious PowerShell Command Line.
- Possible Azure Synapse Integration Runtime exploitation.
Microsoft Sentinel customers can use the following queries based on Microsoft Defender for Endpoint signatures to identify suspicious behavior leveraging this vulnerability.
- Microsoft Defender for Endpoint (MDE) signatures for Azure Synapse pipelines and Azure Data Factory: This query looks for Microsoft Defender for Endpoint detections related to the remote command execution attempts on Azure IR with Managed VNet or SHIR. In Microsoft Sentinel, the SecurityAlerts table includes the name of the impacted device. Additionally, this query joins the DeviceInfo table to connect other information such as device group, IP address, signed in users, and others allowing analysts using Microsoft Sentinel to have more context related to the alert.
- Possible command injection attempts against Azure Integration Runtimes: This hunting query looks for potential command injection attempts via the vulnerable third-party driver against Azure IR with Managed VNet or SHIR processes as well as post-exploitation activity based on process execution and command line activity.
Customer Recommendations and Additional Support
To ensure that your resources receive the necessary security updates, customers using Azure Data Factory with Self-hosted IRs (SHIRs) with auto-update turned off must update their SHIRs to the latest version (5.17.8154.2). Customers can download the latest version here. These customers were also notified of this guidance through Service Health (Tracking ID: MLC3-LD0) in the Azure Portal.
No further action is required of customers using SHIRs with auto-update enabled or customers using Azure IRs.
Customers can read more about the updates made to Self-hosted IR for this vulnerability in the release notes.
For additional protection, Microsoft recommends configuring Synapse workspaces with a Managed Virtual Network which provides better compute and network isolation. Customers using Azure Data Factory can enable Azure integration runtimes with a Managed Virtual Network. Microsoft is continually taking steps to apply additional safeguards to harden the Azure Data Factory and Azure Synapse Analytics platforms and protect our customers.
Ongoing Efforts to Protect Customers
While Microsoft applied the necessary mitigations for the vulnerability that Orca Security reported, we continue to invest engineering effort to ensure that customers of Azure Data Factory and Synapse pipeline workloads running in our cloud are secure and trustworthy. Our ongoing efforts include:
- Continuing to work with our third-party driver vendors to ensure that all updates meet our security standards; sharing with our third-party vendors our security tools and techniques for ensuring a secure footprint
- Ensuring that Cloud processes and workloads, including third-party data connectors, run in a Zero Trust architecture that advance cross tenant isolation. Specifically, we are implementing virtualization of third-party connector execution to achieve per-tenant isolation.
- Proactively monitoring the broader footprint of Microsoft services that leverage 3rd party connectors.
- Continued investment in monitoring and detections for proactive alerting, notification, and accelerated mitigation.
Please visit our Security Advisory for further details on our ongoing effort to address this issue: ADV220001.
We want to acknowledge Orca Security for reporting this vulnerability. We appreciate their continues partnership and collaboration. We encourage all researchers to work with vendors under Coordinated Vulnerability Disclosure (CVD) and abide by the terms and conditions in the Microsoft Bug Bounty Program to avoid impacting customer data while conducting security research.
Additional Resources
- Details on this CVE: CVE-2022-29972
- Microsoft Security Advisory – Defense in depth measures for Azure Data Factory and Azure Synapse pipeline: ADV220001
- Orca Security Blog
The MSRC Team
6-14-2022 - Blog post updated to reflect the steps taken across all IR types and acknowledgements.