This post relates to an unpublished ‘feature’ of Microsoft Azure HDInsight Linux clusters that is misconfigured such that it allows users to obtain root access to clusters without having knowledge of the ‘admin’ account name or password via a web console.
I originally raised this with Microsoft Support around the end of October / beginning of November 2016. Initially, support informed me that they had discussed it with the product team and that the security issue that I was reporting was not a security issue because:
- The security boundary of HDInsight is the Virtual Network (VNET) and
- The clusters are only intended for single user tenancy (ironically a MSFT Cloud Data Solution Architect recently said to me that HDInsight fully supports multiple users – which I guess is sort of true now with secure clusters being in preview).
Eventually they agreed that it was indeed an issue and disabled the feature on all new clusters as an interim measure.
What is the issue?
An Azure HDInsight Linux cluster consists of head, worker and zookeeper nodes – these nodes are Azure VMs, although the VMs are not visible nor can the individual VMs be managed in the Azure Portal you can SSH to the cluster nodes.
When you provision a cluster you are prompted to set to credentials:
- One that will be used for the Ambari web interface – which you can login to over HTTPS and a <cluster name>.azurehdinsight.net domain.
- The other for a local account that will be created on ALL nodes in the cluster which you can then use to SSH to the cluster ssh <user>@<cluster name>-ssh.azurehdinsight.net
The SSH account by default has passwordless sudo – that is you can run sudo su and become root without being prompted for your password.
One of the packages that is installed when you provision a HDInsight cluster is hdinsight-webssh running apt-cache show hdinsight-webssh shows us that it is a Microsoft package (there are other Microsoft HDInsight packages they are all prefixed with hdinsight-):
root@hn0-vjt2-h:/home/dhadmin# apt-cache search hdinsight-
hdinsight-livy - HdInsight livy install
hdinsight-startupagent - HdInsight Startupagent
hdinsight-zeppelin - HdInsight zeppelin sample notebooks
hdinsight-webssh - HDInsight browser terminal emulation
hdinsight-credentialservice - HdInsight datalake credential service
root@hn0-vjt1-h:~# apt-cache show hdinsight-webssh
Status: install ok installed
Depends: default-jre | java6-runtime
Description: HDInsight browser terminal emulation
Running netstat you can see that there is a nodejs based web terminal running and listening on port TCPv6 port 3000:
root@hn0-vjt1-h:~# netstat -atunlp | grep node
tcp6 0 0 :::3000 :::* LISTEN 1361/node
root@hn0-vjt1-h:~# curl -L ':::3000'
<!DOCTYPE html><html><head><meta charset="utf-8"><title>webssh</title><link rel="stylesheet" type="text/css" href="webssh.min.css"><script src="./soc
ket.io/socket.io.js"></script><script src="./socket.io-stream.js"></script><script src="./terminal.js"></script></head><body><div id="modal-alert" class="modal"><div class="modal-content"><h2>This page is no longer controlling the
terminal</h2><br><p>It looks like another page has requested the control of this session. For this reason, this page no longer has access to it.</p><br><button id="request_new_session_btn">Request a new session</button></div></
class="identifier">3c6a307a-2bc1-44ef-b620-fe1e6c00e489</span><span id="sessionId" class="identifier">603cae0b-31c0-41a5-9da9-e523889d56b5</span><span id="w" class="charWidth">X</span><pre id="terminaljs" class="terminaljs" data-
columns="80" data-rows="10"></pre><script src="./webssh.min.js"></script></body></html>
If you run
ps -fuax | grep webssh
you will see the process (which incidentally also runs as root!).
The configuration for the service/application is here:
It looks like that a number of python scripts are run when you provision a cluster to start ambari, configure hive etc. one of which is to start this websshd service with /opt/startup_scripts/startup_webssh.py
Impact of the issue
The issue cannot be easily exploited by an external attacker e.g. one that does not already have access to infrastructure in the Azure Virtual Network (VNET) that the HDInsight cluster resides in. Such an external attacker would first need to gain access to (doesn’t need to be a privileged account) on any other system hosted in the same VNET and from this point they can easily gain root access on the HDInsight cluster by simply browsing to http://
<clusternodeipaddress>:3000 which would automatically give them a web based shell as the user that has passwordless sudo without entering any username or password.
However, since the default NSG rules allow connectivity within a VNET (as opposed to a default deny that requires all traffic to be explicitly allowed) this makes it easier for an attacker to extend their reach.
Another possibility is that an external attacker would need to find a vulnerability in the proxy servers and/or the various web interfaces that are accessible via the proxies.
In the case of a malicious user who has authorised access to say an application or web server, they would be able to take advantage of the misconfiguration to obtain root access to the HDInsight cluster as described above.
In either case an external attacker or malicious user can then use the root access to exfiltrate data, plant malicious software etc.
Microsoft have since disabled the service (although the last time I checked back in December 2016 the package is still installed but the service is not running, nor is there a systemd unit file installed.
Microsoft didn’t explain why the package is installed in the first place but I can only assume it was added as a convenience when the product team were developing or testing.
Browser based terminals are problematic when it comes to security but it’s worse when the endpoint is
- Performs no authentication
- Drops you in as a user that has passwordless sudo
As an added measure you can disable passwordless sudo for the admin account – which probably shouldn’t be enabled anyway.