BeeGFS Security

BeeGFS is increasing in popularity, but like other popular parallel filesystems used in the HPC world it can introduce some rather significant security issues into your environment if it is not well configured. In recent months some rather good changes have been made to the BeeGFS code base which have had a really positive impact on the general security architecture of BeeGFS and do away with some of the flaws which existed previously. But, it is fair to say, there are a couple things that are quite easy to get wrong, even on current BeeGFS installs, which can leave your installation presenting a pretty significant risk to your cluster. This blog post is written to highlight some of these and to offer some guidance in securing your installation.

Parallel filesystems are always of great interest in our work at HPCsec, partly because we’ve found rather significant flaws in every parallel filesystem we have ever looked at; but also because, since they are often mounted by multiple clusters, they can (depending on how they are used) present a great platform for an adversary to move from one system to another.

The most prevalent current BeeGFS security configuration issues can be resolved by ensuring the following:

  • Ensure that you are using a connAuthFile
  • Ensure the permissions on your connAuthFile are set securely
  • Ensure that you are mounting your BeeGFS filesystem “nosuid”
  • Do not rely on the connNetFilterFile.conf or connInterfacesFile.conf to control access

The rest of this post elaborates more on these four points.

Background

Back in 2019 HPCsec took a look at BeeGFS (version 7.1.3 was current at the time) and found some rather alarming flaws in a default installation which allowed for anyone on a node mounting BeeGFS to elevate their privileges and become root. HPCsec reported this to ThinkParQ and it was assigned CVE-2019-15897 (you’ll find our advisory here: https://www.hpcsec.com/2019/12/04/cve-2019-15897/). Below is a video of us exploiting this vulnerability in order elevate privileges and become root on a system using BeeGFS:

Vulnerability in BeeGFS being exploited in order to escalate privileges to root

Whilst there are a couple of quite interesting elements to this exploit we haven’t, and do not intend to, share the specific details of this vulnerability publicly for various reasons. What was really interesting to us though, was that a mechanism to fix this vulnerability (and some other vulnerabilities too) was already present in BeeGFS, it was just not part of a default install and not something that would typically be stumbled upon. That mechanism was the use of the “connAuthFile” which, when set up correctly, was very effective in preventing this issue from being exploitable. Despite its presence it was rarely used and as a result most BeeGFS installations were vulnerable to this issue.

However, in releases 7.3.2 and 7.2.8 of BeeGFS (October 2022) something significant changed, the default install required you to configure a connAuthFile or to elect to disable authentication. This appeared very prominently in the release notes for both v7.3.2 and v7.2.8: [https://doc.beegfs.io/7.3.2/release_notes.html] [https://doc.beegfs.io/7.2.8/release_notes.html] and BeeGFS would simply not start unless you had chosen one or the other. These changes gave us a kick to put together a bit more context and information together on BeeGFS security and is the reason we put this post together with the 4 bullet points we think rectify some of the most prevalent security configuration issues we have seen with BeeGFS installs.

Ensure that you are using a connAuthFile

The connAuthFile is a file which essentially contains a shared key. For a BeeGFS cluster to function every node must have access to the same key otherwise the node will not be permitted to talk to the BeeGFS services. You configure your connAuthFile within your BeeGFS configuration file (typically /etc/beegfs/beegfs.conf) with an entry such as this:

connAuthFile=/etc/beegfs/connAuthFile

This file must be present on all BeeGFS nodes as well as any clients that will mount BeeGFS. The contents of the connAuthFile can be whatever you like, the contents of the file are split in half and two 32-bit hashes are generated, one from the first half and one from the second half of the contents. These both come together to form the 64-bit connection authentication hash. Any node which wishes to participate in the BeeGFS communications must establish its connections using this key. You can elect not to use any authentication instead by setting the “connDisableAuthentication” value to “true” in your BeeGFS configuration, but this is of course not recommended for any production system as it can lead to the privilege escalation shown in the video above.

Ensure the permissions on your connAuthFile are set securely

The thing that is most critical with this connAuthFile is that it should only be readable by root. If any other user can read this file then they can perform operations which are likely to allow them to gain root access to your cluster. The following are recommended settings:

# chown root:root /etc/beegfs/connAuthFile
# chmod 400 /etc/beegfs/connAuthFile

In BeeGFS versions prior to 7.3.2 and 7.2.8 setting these permissions would prevent some aspects of “beegfs-ctl” from functioning as a non-root user because the key could not be read from the connAuthFile. However, in later versions beegfs-ctl has been made setuid so has the ability to read this file and functions as expected and therefore all functionality works for all users. There is always risk associated with setuid files so, despite this, HPCsec’s recommendations would be to unset the setuid bit (i.e “chmod 0755 beegfs-ctl”) unless there are some really specific reasons that non-root users need to run beegfs-ctl.

Ensure that you are mounting your BeeGFS filesystem “nosuid”

If you paid attention to the video you’ll have seen that the exploit created a setuid file and then copied korn shell into it (we used ksh because it is widely available and shells like bash will not run setuid). In doing so we now have a shell that when run elevates our privileges to root. If, however, BeeGFS were mounted with the “nosuid” flag this would prevent this particular route of exploitation from being possible. As of v7.3.1 BeeGFS mounts “nosuid” by default, however, should you wish to be doubly sure that this is the case, or you are running an older version of BeeGFS, then you can add the following to your /etc/beegfs/beegfs-mounts.conf (or your /etc/fstab):

/mnt/beegfs /etc/beegfs/beegfs-client.conf beegfs nosuid

The principle of mounting nosuid is also something that HPCsec would advise is applied to any other shared/parallel filesystems that you use within your environment. If you have a number of clusters or supercomputers that mount the same filesystem (e.g. users home directories or scratch space) a flaw on one system that allows for a setuid binary to be created suddenly provides a mechanism for elevating privileges on any other systems that mounts that same filesystem if it is not mounted nosuid.

connNetFilterFile.conf + connInterfacesFile.conf

The connNetFilterFile and connInterfacesFile are two configuration files that control what network ranges are able to connect to the BeeGFS instance and what interfaces BeeGFS services should register respectively. These do perform the function required of them as long as everyone playing follows the rules; unfortunately that is not how adversaries chose to play. If we take the connNetFilterFile as an example, in our BeeGFS test cluster if we run “beegfs-check-servers” we get the following output:

bash-4.2$ beegfs-check-servers 
Management
==========
node01 [ID: 1]: reachable at 10.1.33.202:8008 (protocol: TCP)

Metadata
==========
node02 [ID: 2]: reachable at 10.1.33.203:8005 (protocol: TCP)

Storage
==========
node03 [ID: 3]: reachable at 10.1.33.204:8003 (protocol: TCP)

If we deny the network range that our host is in, by adding it to the connNetFilterFile, we get no results because we’re now on a denied range and so the management server will not share where the other servers exist with us. However, that hasn’t actually prevented us from communicating with the metadata or storage servers it has simply prevented the management server from sharing that information with us. If you’re following the process flow that legitimate BeeGFS communication would then it might appear that these configuration files restrict access but in reality they have not. In fact the exploit shown in the video above works perfectly well even if these configuration files are in place and set to restrict access from our IP address/range.

These files are not intended for security purposes and so should not be used or relied on for security purposes. Instead use the connAuthFile, this not only solves the security issues that you are likely to be trying to solve with the connNetFilterFile and connInterfacesFile but also plugs a number of other holes and is essential if you want any form of security in your BeeGFS cluster. No harm in using these files in addition to a connAuthFile, just don’t use them in place of them.

It is entirely likely that you did not even know these files existed, in which case you need not worry about this section. But if you have used these files in order to control access then hopefully this has given a bit of context as to why these files alone are not sufficient to secure a BeeGFS install and why a connAuthFile remains necessary.

Summary

If you follow these four bullet points then your BeeGFS installation will not be affected by some of the most common security configuration issues we typically see with BeeGFS deployments that can present a significant risk to a cluster:

  • Ensure that you are using a connAuthFile
  • Ensure the permissions on your connAuthFile are set securely
  • Ensure that you are mounting your BeeGFS filesystem “nosuid”
  • Do not rely on the connNetFilterFile.conf or connInterfacesFile.conf to control access

These controls can all be implemented without a need to update to the latest version of BeeGFS, so even if you are running an older version of BeeGFS prior to some of these settings becoming default options there is no reason that your installation cannot benefit from the security that they add.