Management Interfaces - Attack Surface Hidden in Plain Sight
Table of Contents
by:
Lari Huttunen
A management interface, who is it for? #
Modern web-based management interfaces help with the economy of scale. If you are a software vendor making a solution, supporting it is easier with clearly defined UI options rather than debugging obscure configuration file parameters. If you are an end-user, a management interface is there to make life easier for you as well.
Having a management interface helps you:
- deploy the solution
- make complex changes to it
- generate management level reporting for the KPI.
These features often become tender items and a vendor will find itself in a position where developing a management interface web UI is a must have instead of a nice to have. Too often features are implemented in software through a tick box comparison, since the rationale is that we must have them since our competitor has them. It doesn’t really matter, whether the features actually serve the customer and their business function or not. From the customer’s perspective the scalable solution might actually be a management REST API, but many of the solutions still heavily rely on this manual management model instead – which doesn’t really help with scalability issues at the customer end at all.
Is it for your eyes only? #
As the name suggests, a management interface is used to configure and maintain a device or a solution. Traditionally, this has meant console access either remotely or through a local terminal. Nowadays, a management interface often spells a web UI. The web is there mostly for the ease of use but also to be able to visualize and manage complex configurations which over a text-based command line interface might be quite challenging.
In security conscious applications, the management interface has been segregated from the user facing services. The simplest way to do this is to make the management interface listen to a different port than the main application. This way, the management interface can be firewalled or access controlled even if the application or solution has only one public-facing IP. Too often, default configurations for (physical) appliances make it too easy to misconfigure the solution so that the management interface is listening on the WAN interface instead of the LAN one.
In either case, separating the management function from the rest of the business functions is a must have and helps reduce the external attack surface of the application, since should a vulnerability be discovered in the interface it is not exposed to everyone and their grandma to talk to.
Attack Surface vs Attack Vector #
A management interface exposed to the whole Internet represents an attack surface which often is a softer target than the user facing part of a given application or solution. As stated above, this attack surface is usually either ignored or left open by accident. Often, the interface is exposed to the Internet because it is not easy to restrict the management interface to a separate port or network interface. Furthermore, even if the interface is sitting on a separate port, there may not be an easy solution to expose it only to specific source IPs which are dedicated for management access. Be as it may, the actual reason for this public exposure seems to be ease of access for the administrator – at the cost of operational security. Furthermore, the administrators may even be outsourced which further complicates the issue, since configuration management rests with them and not with you.
Newly discovered or residual vulnerabilities can turn an attack surface into an attack vector. Too often, it is only then when people take notice. Even when a given vector is mitigated, nobody seems to care that the attack surface remains in place – ready for the next round of exploitation. The vendor stance often is that the interface is not meant to be exposed to the Internet. This is a poor excuse not to invest in something that would stand the test of time and exposure to the criminal elements.
Management Interface Vulnerabilities #
In the past couple of years, management interfaces have faced scrutiny from security researchers and as a result a plethora of vulnerabilities have surfaced. A common misconception seems to be that if a solution is a security solution its code base must be secure as well. Let’s take a firewall for example. Its user-facing functions are most likely secure, but its soft underbelly is the web-based management interface, which I will exemplify below.
I mean a Java application that controls a firewall is likely to have vulnerabilities in it in the same manner as any other Java application. Moreover, such a management application needs privileged access to the solution which in turn means that a vulnerability discovered in it is likely to have serious consequences. Too often, we see directory traversal vulnerabilities in these applications which means that the developers are not really focused on security.
How to Dodge the Bullet #
The best mitigation for an on-premises solution is to use a dedicated management network with a dedicated physical interface through which the solution is managed. This is fine and dandy if we are talking about a physical appliance in the first place that resides on your premises. More often than not a solution is virtualized and resides in the cloud, which means that the management network will have to be virtual as well.
A common way to protect a web-based management interface is to use a Web Application Firewall (WAF) in front of the solution, but usually this is just a sign of poor implementation of the solution where the user and management functions have not been properly separated. In any case, an often overlooked aspect of these bastion hosts or middle boxes is the lack of proper AAA. Making sure that the management connections and actions are actually properly logged and the logs are duplicated outside the bastion host is a good measure for the worst case scenario – that the admin account is compromised on the client side.
Attack Vector Examples from Recent History #
Below, I will detail some of the recent public exposure related to management interfaces of various solutions. The sad fact seems to be that no one seems to note this exposure until either a high profile breach has occurred through them or a critical RCE is disclosed to the public – or both. Three of the examples represent a traditional appliance model (F5, Sophos, Zyxel) and two of the examples a modern API interface (Kubernetes and WinRM).
F5 TMOS vs. TMUI #
F5 produces middle box security products, which help organizations manage their web applications through load balancing, reverse proxying or by providing a web application firewall. These complex applications of course also call for complex management interfaces, which have been subject to severe vulnerabilities over the past two years.
The most recent attack vector disclosed in May 2022 is CVE-2022-1388, which allows remote code execution on the management interface. Above, I have visualized the total number of publicly reachable F5 TMOS platforms (33900) and the total number of publicly reachable F5 TMUI management interfaces (1650). 4.9% may not sound bad, but these middle box applications are something that these organizations depend on to protect their API endpoints and web applications, i.e. the core of their online business.
Sophos Firewall #
Sophos produces firewalls which ship with a web portal for user access, as well as admin access. This attack surface was turned into an attack vector through CVE-2022-1040, which allows an unauthenticated user to execute code remotely on the firewall. Even if these vulnerabilities are fixed, Sophos does recommend its users not to expose these portals to the WAN interface.
The total number of exposed management interfaces in May 2022 was roughly 160k, where approximately 1/3 of these devices were located in India and the US.
Zyxel Firewall #
Zyxel Networks is the manufacturer of Zyxel firewalls. CVE-2022-30525 turned the zero touch provisioning feature into unauthenticated remote code execution via OS command injection through the CGI binary. To me this type of vulnerability smells strongly of the 1990s, since who is using CGI any more in this day and age especially for security device management?
For Zyxel firewalls, the attack vector is most prominent in France and Italy, which accounted for roughly half of the 16k vulnerable devices in May 2022.
Kubernetes API #
The Kubernetes API lets you query and manipulate the state of objects in Kubernetes. The core of Kubernetes’ control plane is the API server and the HTTP API that it exposes. Users, the different parts of your cluster, and external components all communicate with one another through the API server.
The statement above defines the Kubernetes API as a very powerful management interface, since if poorly configured, a malicious third party may manipulate your running workloads or generate their own. That is why it is recommended that you do not expose this API directly to the Internet to provide some defense-in-depth for the management interface of your Kubernetes cluster.
PaloAlto Unit 42 has written an in-depth analysis of the unsecured Kubernetes clusters, which can be used as an attack vector and luckily only numbered at 6k API servers.
Windows Remote Management Service, WinRM #
Another remote management interface with a massive attack surface globally is WinRM. According to Rapid7 research, it is a SOAP interface which you can use to manage the box remotely. One mitigating factor may be that by default the service may be in a non-functional state which limits the attack vector even if the attack surface is present.
Roughly 1.8 million winrm interfaces are exposed on a monthly basis to the whole Internet. The biggest geolocations for publicly exposed winrm are USA (~600k), China (~500k) and Hong Kong (~180k). In terms of an attack vector, CVE-2021-31166, a remote code execution vulnerability in WinRM exemplifies why this interface must not be directly exposed to the Internet.
Wrapping it Up #
By now, it should be evident that an exposed attack surface will in time be turned into an attack vector and the only practical way to mitigate that attack surface is to control access to it. In this post I’ve focused on management APIs which in my opinion form a dangerous attack surface hidden in plain sight. Too often, sysadmins seem to be led astray by thinking that an interface is secure since it has been developed by an information security company. Given the number of vulnerabilities in security products over the past year(s) one should not be lulled into complacency by authority alone. Letting everyone and their grandma interact with your management API is not a good practice in the long run.