For the last few years ransomware has been one of the greatest cyber security threats to most organizations. Hugely evolved from the days of WannaCry and Petya/NotPetya, ransomware is no longer a pre-programmed piece of malware, it’s a human operated campaign with the goal of cyber extortion. Humans are not pre-programmed pieces of code, so effective defences are difficult. How an attack manifests itself on your network may look quite different to another attack even if conducted by the same operators. These are skilled operators who have access to tooling and infrastructure through affiliate programs. It’s a world away from the ransomware campaigns that first caught the world’s attention in 2017.
The Current State of Ransomware
Predominantly attacks have hit Windows environments with access often gained by Initial Access Brokers (IAB) who pass on their access (for a fee) to the ransomware groups to utilise. Typically they will be looking to achieve a full compromise of a domain and utilise that to exfiltrate and encrypt sensitive data. If, having encrypted critical data, the victim organisation is unwilling to pay a ransom in order to recover their data the second stage of extortion kicks in (hence the term double extortion). At this stage the ransomware group will typically post data exfiltrated online and demand payment that the data is not published. If you’re holding very sensitive, knowing that you have backups is no help against this threat.
Ultimately ransomware is devastating organisations across the world and making criminals a lot of money. The key to this success is the fact that adversaries have landed on a model whereby they can monetise an attack on any organisation. They don’t have to worry about finding a buyer for the data they steal, they have a buyer lined up already, that buyer is the victim organisation.
Ransomware and Linux
At HPCsec, we follow ransomware attacks extremely closely and hold a huge amount of data relating to attacks which have been conducted. For the last few years there has been a theme across most ransomware groups of retooling in order to support Linux systems. Predominantly this was in recognition of the fact that many organisations ran VMware ESXi, so targeting ESXi servers was a great way of extorting a business. Whilst a lot of the tooling we see off the back of incidents specifically targets .vmdk and other virtual machine files this is simply a preference rather than a limitation of the tooling. Ultimately the tooling has the ability to run on any Linux system that has been compromised. HPCsec first analysed RansomEXX’s Linux variant back in 2020 and like most groups their tooling is compiled on a per victim basis and can be customised to the specific need. It has all of the capabilities to cause real damage to HPC systems.
What’s Changing
For a while human operated ransomware campaigns very much took the approach of “compromise the entire domain and encrypt all of the systems”. This required a high level of access (Domain Admin typically). But as things evolved into a double extortion model where accessing sensitive data rather than mass disruption became key we started to see a change in modes of operation. We started to see an awful lot more single system compromises, such as data taken from fileservers without a particularly widespread domain compromise and source code repositories being lifted.
The most prominent entry point to an organisation has remained via email phishing. With the vast majority of the World using Windows systems an attack will typically start on a Windows PC and inherently that leads to a focus on the Windows domain – but when that is not the case is when things are starting to look different. And this is becoming increasingly not the case. Increasingly we are seeing exploits being used in order to gain initial access to systems and that is presenting a different picture to ransomware operators, they’re seeing different systems and different opportunities. Already this year we’ve seen a campaign dubbed ESXiArgs which mass targeted ESXi servers, we’ve seen groups demonstrating encryption of a RTU and a general theme of ICS/OT ransomware. Not specifically ransomware but we’ve seen satellites being bricked and other technologies being targeted too.
Ultimately the ransomware operators will work with what access their Initial Access Brokers provide. Historically Windows systems and domains will have been the natural target, but right now we know that they mostly have the tooling to hit anything and we know that they are increasingly aware of their environments. Let’s not forget that these are human operators, this is not commodity pre-programmed malware. Right now they have all of the capabilities and tooling to cause some serious damage to HPC environments.
Most of this post has been painting the picture and providing a general overview of ransomware and how it has evolved in recent years. In future years we expect to see a more focused approach for certain organizations and consequently the compromise of higher value targets – ICS, OT, Network equipment, supercompters etc. the type of systems where disruption to their operation is significant, i.e. fewer but higher value targets. This will likely still be fed by opportunist attacks for a while, very much taking advantage of the fact that cyber extortion is now a familiar concept and part of day to day cyber security.
So what about HPC clusters, what should you consider?
The following are some areas to be mindful of in securing your HPC environment from ransomware:
Authentication and Trust Relationships
If your HPC clusters or supercomputers are tied in with your Windows domain (perhaps utilisting kerberos) then that trust relationship is a risk. A compromise of your Windows domain could easily result in a compromise of your HPC environment. If an adversary has not done their homework or has misunderstood your AD groups then you may well see failed login attempts for users who typically wouldn’t use the HPC cluster. Additionally you may see both failed and successful logins from network ranges that HPC users would not typically log in from – these may be useful indicators.
We should expect trust relationships to be exploited if such systems are compromised. We saw this in 2020 when several academic supercomputers across Europe were compromised (fortunately only to mine cryptocurrency). Depending on the environment this could lead to multiple clusters being compromised.
Internet Facing Systems
If your HPC systems are internet facing then they will already be being targeted on a constant basis. Being across the system hardening, including security updates is key. It’s also essential to consider the services which are exposed and how you monitor and permit/deny user access. Most breaches of Linux based internet facing systems are down to the use of weak passwords, so disabling password based authentication in preference of key based authentication is a great start. There are both freely available and commercial MFA options for Linux which add another layer of defence.
There are also some usage scenarios to consider with internet facing systems – Are there particular countries your users would not log in from? Are there impossible travel scenarios that could be considered (i.e. user logged in from France and then moments later logged in from India)? These will not always be perfect, but can help identify concerning behaviour. HPCsec provides a freely available list of known bad IPs targeting Linux systems via our threat exchange, it is update hourly. We’d certainly recommend acting if you see successful authentication from any of those IP addresses. But be mindful that the internet is noisy, don’t go overwhelming yourself in alerts, that can be as ineffective as doing nothing.
System Architecture
Unfortunately the vast majority of HPC vendors do not adequately segment the administration of their systems from the users. Whilst re-imaging login and compute nodes in most clusters might be considered a temporary annoyance – if administration nodes or other periphery systems such as UPS’s, network switches, cooling systems etc. were compromised this would likely constitute a very significant system rebuild. Preventing access from user nodes to services that are not required is a great start. But if you don’t know what the attack surface for your system looks like in the first place then you should probably start there by getting someone who knows HPC security to provide this insight. At least then you know what you are working with.
File Systems
Parallel file systems are fast, supercomputers are powerful. As a result, encryption of files can take place comparatively more rapidly than on a regular Linux system meaning the damage can be done a lot more quickly. If there are monitoring options then consider whether you can leverage this. Frequently HPC systems are multi-user environments. As such user home directories and scratch space make it relatively easy for operators to identify sensitive files and data for exfiltration.
Key files that ransomware operators will likely take from compromised Linux systems will include shadow files and keys (ssh private keys). If the extent of a compromise is not fully understood or cleaned up after and these files are shared on the dark web then further compromises may take place.
Data
There is often not a need for users to be able to establish outbound connections from a HPC system. Blocking such access means that if a ransomware operator wishes to exfiltrate data they will be unable to do it directly.
Backups are not going to help contain any data that an adversary has taken from your environment, but they can ensure that you have access to your data should it be encrypted. Backups are one thing that we’ve seen ransomware operators targeting – of course, if they encrypt your backups the likelihood of you paying a ransom vastly increases. So keeping these backups offline or at the very least behind some other authentication perhaps in a read-only manner.
Threat Model
Ultimately the best defences will come from properly understanding the threat model of your environment, knowing what it looks like to an adversary, and building out the right defensive controls. What works in one environment might be irrelevant in another. Please don’t rely on your support contract to cover system security.
At HPCsec securing HPC systems is what we do, it’s all we do, and we do it extremely well. We have tooling to achieve everything above, so if the security of your HPC systems is a concern then you can always get in touch. We’re more than happy to chat without any commitment.