Microsoft said it has discovered a new attack vector against Kubernetes workloads that involves Kubeflow, a machine learning toolkit for Kubernetes. According to Yossi Weizman, a security researcher with Microsoft's Azure Security Center, the attacks have been occurring since April, 2020, and they aim to install cryptojacking malware on Kubernetes clusters running Kubeflow instances exposed to the internet.
The researcher said the company has observed attacks “on tens of Kubernetes clusters” running Kubeflow.
“Nodes that are used for ML tasks are often relatively powerful, and in some cases include GPUs. This fact makes Kubernetes clusters that are used for ML tasks a perfect target for crypto mining campaigns, which was the aim of this attack,” according to the report.
In April, the researchers discovered a suspicious image (ddsfdfsaadfs/dfsdf:99) from a public repository deployed on many clusters. The analysis revealed that this image contained an XMRIG miner, a Monero cryptocurrency mining tool. Microsoft says that besides above mentioned image, the public repository also contained several more images with different mining configuration.
The researchers believe that the entry point for the attacks are misconfigured Kubeflow instances. Users often change the Kubeflow default settings for convenience purposes, which exposes the toolkit's admin panel on the internet. By default, the Kubeflow management panel is exposed only internally and accessible from inside the Kubernetes cluster.
“In some cases, users modify the setting of the Istio Service to Load-Balancer which exposes the Service (istio-ingressgateway in the namespace istio-system) to the Internet. We believe that some users chose to do it for convenience: without this action, accessing to the dashboard requires tunneling through the Kubernetes API server and isn’t direct,” Microsoft explained.
“By exposing the Service to the Internet, users can access to the dashboard directly. However, this operation enables insecure access to the Kubeflow dashboard, which allows anyone to perform operations in Kubeflow, including deploying new containers in the cluster,” the company added.
In observed attacks the threat actors used exposed dashbords to gain access to the cluster and install the cryptocurrency miner.
“Azure Security Center has detected multiple campaigns against Kubernetes clusters in the past that have a similar access vector: an exposed service to the internet. However, this is the first time that we have identified an attack that targets Kubeflow environments specifically,” Microsoft said.