Spectrum Scale Exploit Analysis

An Analysis and a Check for an IBM Spectrum Scale Vulnerability

Recently some exploit code appeared on line for a vulnerability in Spectrum Scale (GPFS) v5.0.3. This vulnerability is one of a series of vulnerabilities which have been made exploitable as a result of IBM choosing to set the the setUID bit on a number of root owned binaries within the /usr/lpp/mmfs/bin directory. Like similar past vulnerabilities, this vulnerability also results in an ordinary user being able to gain full root access to a system using Spectrum Scale.

This article shares an analysis of the vulnerability and also provides a script that can be run in order to check whether your systems are vulnerable.

IBM rarely provide much useful information about vulnerabilities affecting their technologies. Consequently it can be quite difficult to ensure that systems are secure without performing some time consuming analysis… which is what we’ve done and hope is useful:

Version Information

We won’t share the exploit code here because it can be disruptive on a production system unless you clean up after it. Nevertheless it was a very elegantly written and reliable piece of exploit code.

We believe that it was an exploit for CVE-2020-4273 (or possibly CVE-2019-4558), both of which have been fixed by IBM. If you are not patched against these vulnerabilities then you should do so urgently. We have been able to confirm that the exploit code works against Spectrum Scale v5.0.3 and does NOT work against v5.0.5. It does also affect the 4.2.x branch. Our belief is that a patch for this vulnerability appeared in v5.0.4.3 and v4.2.3.21.

The Technical Detail

For those that are interested, this exploit takes advantage of the fact that an fopen() call in the affected binaries truncates its input and prepends the data held in the GPFS_CMD_TRACING_LOG environment variable. Shove enough data in that environment variable and it’ll truncate it stripping off the intended destination and giving you control over where it writes its file.

If the GPFS_CMD_TRACING_LOG environment variable is set then you’ll see an open call along these lines (where xxxxx is the value contained within the environment variable). If no environment variable is set then this open call will not happen:

open("xxxxx/mmfs_cmd.tschfileset.8791", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 ENOTDIR (Not a directory)

With a specially crafted environment variable you can cause the intended path to be truncated and result in the following open() call happening:

open("////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////tmp/hpcsec", O_WRONLY|O_CREAT|O_TRUNC, 0666)

Due to the use of setUID bits on various Spectrum Scale related binaries the file created as a result of this call is root owned and writable by the user. It therefore provides the platform for the user to escalate their privileges to root (the detail of which we won’t go into here).

-rw-rw-r--.  1 root     tes1          50 Jul  3 15:47 hpcsec

As we’ve already mentioned, with the setUID bits removed from the binaries in /usr/lpp/mmfs/bin/ there is no opportunity for this vulnerability to be exploited. In fact (other than those affecting the optional GUI) if you remove the setUID bits from all binaries in /usr/lpp/mmfs/bin, as we recommended back in 2016, you’d have protected yourself from all notable Spectrum Scale vulnerabilities that have ever existed (including this one) and likely most that will exist in the future too. As I’m not aware of any reason for these setUID bits to be set, they appear to do nothing but open up security holes I cannot recommend strongly enough that you remove those setUID bits.

Checking for the Vulnerability

We have provided a script which is freely downloadable and can be run to check whether your system is vulnerable: https://files.hpcsec.com/utilities/check-cve-2020-4273.sh

Whilst the script does clean up after itself you may be left with an empty root owned file “/tmp/hpcsec” on your system which you’ll need to manually remove.

# https://files.hpcsec.com/utilities/check-cve-2020-4273.sh
# A script to check for vulnerability to CVE-2020-4273
# (or possibly CVE-2019-4558). The script checks for a call
# to setresuid() which is one ofthe modifications IBM made
# in order to patch thevulnerability
# you should run this script as a non-root user
# if you are vulnerable it will create an empty root owned
# file named /tmp/hpcsec which you will need to manually
# remove
# We believe this script to be very safe, however, use at
# your own risk.
# Usage:
# $ bash check-cve-2020-4273.sh

if [[ $EUID -eq 0 ]]; then
   echo "This script must be run as a non root user !" 1>&2
   exit 1
export GPFS_CMD_TRACING_LOC=`perl -e 'print "/" x 243'`/tmp/hpcsec
# first run without strace as it messes with file permissions
/usr/lpp/mmfs/bin/tschfileset 2>/dev/null
# run again through strace to capture output

if which strace &>/dev/null; then
        strace -e trace=setresuid -o /tmp/hpcsec.strace /usr/lpp/mmfs/bin/tschfileset 2>/dev/null
        # if strace output includes call to setresuid() then you're probably patched
        if grep setresuid /tmp/hpcsec.strace; then
                echo "[ ] you appear to be patched (good)"
                echo "[X] you appear to not be patched (bad)"
        echo "[?] you do not have strace installed, unable to check whether you are patched (install strace and retry)"


# if you can write to /tmp/hpcsec but don't own it then you're probably vulnerable
if test -w /tmp/hpcsec && !(test -O /tmp/hpcsec); then
        echo "[X] you are vulnerable (very bad)"
        echo "[ ] you appear not to be not vulnerable (good)"

# if you have set UID bits set on any binaries in /usr/lpp/mmfs/bin then you are a fool!
if test -u /usr/lpp/mmfs/bin/tschfileset; then
        echo "[X] set uid bit set on your tschfileset binary (bad)"
        echo "[ ] no set UID bits on your tschfileset binary (good)"

rm -f /tmp/hpcsec 2>/dev/null
rm -f /tmp/hpcsec.strace 2>/dev/null
echo "https://www.hpcsec.com/"

For those interested in a bit more detail on what the script is doing here is snippet of the strace output on a vulnerable and a patched system around where it writes a file to disk:

Vulnerable System

rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
getuid()                                = 0
getuid()                                = 0
brk(NULL)                               = 0x55df15f5c000
brk(0x55df15f7d000)                     = 0x55df15f7d000
brk(NULL)                               = 0x55df15f7d000
open("////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////tmp/hpcsec", O_WRONLY|O_CREAT|O_TRUNC, 0666)
= 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=106075056, ...}) = 0

Patched System

rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
brk(NULL)                               = 0x55c6dcdeb000
brk(0x55c6dce0c000)                     = 0x55c6dce0c000
getuid()                                = 1002
getuid()                                = 1002
getpid()                                = 16615
getuid()                                = 1002
geteuid()                               = 1002
setresuid(-1, 1002, -1)                 = 0
umask(0177177)                          = 002
openat(AT_FDCWD, "////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////tmp/hpcsec", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
setresuid(-1, 1002, -1)                 = 0
umask(002)                              = 0177
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

As you can see, on the patched system there are a few more calls, of particular interest are those to:

  • setresuid()
  • umask()

Essentially what IBM have done in their fix is to set a more restrictive umask on the file created, so that only the owner of the file can read and write to it and nobody else has any permissions. They also ensure that the user ID is set to the real user ID even though the effective user ID may be 0. Both of these combined prevent exploitation of this vulnerability. The truncation of the filename still occurs, however, the controls now in place mean that this can only be leveraged to write a file under your own UID and so no longer facilitates privilege escalation.

We would still love to see IBM remove the setUID bits from all Spectrum Scale binaries. However, you can do this yourself; The following command will identify all such binaries:

ls -l /usr/lpp/mmfs/bin | grep r-s

You can reset the setuid bit for each such file by issuing this command on each file

chmod u-s file

References and Other Information

Our advisory for CVE-2020-4273 with links to IBM’s security bulletin is available here: https://www.hpcsec.com/2020/04/07/cve-2020-4273/

Our script to check whether your system is vulnerable is available here: https://files.hpcsec.com/utilities/check-cve-2020-4273.sh

The information within this article was provided to subscribers of our mailing list several weeks back and our customers had fixes 5 years ago. If the security of your HPC environment is critical then do take a minute to subscribe to our mailing list (https://www.hpcsec.com/hpcsec-mailing-lists/) or to get in touch https://www.hpcsec.com/contact/, we’re always looking for new research partners to collaborate with in enhancing HPC security.

If you have some useful information that has not made its way into this post then let us know and we’ll look to incorporate it.