Ahhh… life in the wild frontier of the internet with it’s pervasive exploits, backdoors, rootkits, and other goodies. It was quite a surprise this morning when I was reading through my daily logging reports and found that I had a process spring up since yesterday’s report telling me that a program named ./r0nin was newly listening to port 1666 on one of my web servers. So aside went the mozilla testing to take a peek as this is a client webserver and, well, I don’t take kindly to tampering of any sort. Besides, it’s been a while since I put my forensic skills to the test.
So let me give you the lowdown and if nothing else, it’ll serve as a reminder to me who, thankfully, has only had to do this once before (damn phpNuke!). Now it’s damn a lazy client who decided running WordPress 1.3alpha2 was a good idea, especially considering all the xmlrpc noise that’s been made lately. How’s that for a quick summary? Read on for the goods.
The first notice was this:
Security Warning: There are modifications for port listening on your machine :
- Opened ports : tcp 0 0 *:1666 *:* LISTEN 24465/r0nin
Well, that sure didn’t look good. So I examined the other email reports (these are coming from rsec, the baby-brother of msec designed for Annvix). The other reports didn’t bring to light anything of interest. My tripwire report showed nothing had changed within the last 24hrs of the stuff I monitor (which doesn’t include the web roots, which I may change). So at least no binaries had been tampered with, which is a good sign.
The next step was to find r0nin on that machine. A quick find / -name r0nin found it in the web root of one of my client’s websites. Now, this client runs a lot of different packages… geeklog, gallery, and wordpress; probably a few other things I didn’t immediately notice. Anyways, in the webroot was r0nin, w00t, uselib24, and dc.txt. The first three were executables, the last a perl script which identifies itself as “Data Cha0s Connect Back Backdoor”.
So now I know which site is at fault… I have the timestamps of the files indicating they were placed there or modified at 15:00, 15:06, 15:05, and 15:05 respectively (all on Sep 25). Time to scrub some logfiles. Doing some manual “surfing” of the logfiles showed me the following entries:
200.164.102.51 - - [25/Sep/2005:14:59:44 -0600] "POST /wp/xmlrpc.php HTTP/1.1" 200 61 "-" "-" 200.164.102.51 - - [25/Sep/2005:14:59:48 -0600] "POST /wp/xmlrpc.php HTTP/1.1" 200 2696 "-" "-" 200.164.102.51 - - [25/Sep/2005:14:59:56 -0600] "POST /wp/xmlrpc.php HTTP/1.1" 200 110 "-" "-" 200.164.102.51 - - [25/Sep/2005:15:00:20 -0600] "POST /wp/xmlrpc.php HTTP/1.1" 200 13 "-" "-" 200.164.102.51 - - [25/Sep/2005:15:00:25 -0600] "POST /wp/xmlrpc.php HTTP/1.1" 200 1145 "-" "-" 200.164.102.51 - - [25/Sep/2005:15:00:36 -0600] "POST /wp/xmlrpc.php HTTP/1.1" 200 65 "-" "-"
Interesting. WordPress has had a recent share of XMLRPC issues so this looked very suspicious, and it hit my timeframe. Grepping in the wp/ directory showed me that the version was 1.3-alpha-2. Looking at the website, 1.5.2 is out so I knew right there that the xmlrpc vulns were the culprit.
I decided to determine whether the executables were compiled on the system or whether they were just uploaded. Unfortunately, with the xmlrpc server, nothing got logged so I had no idea what they did through it. But doing some grepping and strings on the binary showed me they were not built with SSP protection, which indicated they were not compiled on my system. Great news.
So now I’ve determined they got to upload their backdoor files via WordPress’ xmlrpc vulns. The files are owned by apache, and a ps axu showed me that r0nin was running as user apache. Didn’t look like they obtained root this way, so this was a bit of a bootstrap. Obviously r0nin is some daemon to spawn a shell, probably as user apache, on the system. Note to self: see if apache can happily run with a shell set to /bin/false rather than /bin/sh.
I wasn’t too concerned when I saw r0nin listening on port 1666… showed me it wasn’t a priv port (good), and I also firewall the snot our of my servers… I have rules in place to limit inbound and outbound traffic; ie. that machine can’t even open an FTP connection to the outside world. So I was pretty confident they wouldn’t get in that way.
Grepping my kernel logfiles proved me right:
Sep 25 15:00:47 host kernel: Shorewall:net2all:DROP:IN=eth0 OUT= \ MAC=00:50:ba:42:f7:0e:00:0d:bd:81:90:41:08:00 SRC=200.164.102.51 \ DST=[my_ip] LEN=48 TOS=0x00 PREC=0x00 TTL=111 ID=5968 DF PROTO=TCP \ SPT=1379 DPT=1666 WINDOW=64800 RES=0x00 SYN URGP=0
(There were three attempts within a minute of each other… wrapped manually so as not to screw up the viewing output).
Same timeframe, same IP adress, and the destination port (DPT) was 1666, the port r0nin was listening to. Thank you, Shorewall.
Since there was nothing else logged, and nothing else suspicious, I figure the script kiddy moved on after he realized he wasn’t getting in. Poor boy. After looking at all of this and determining that he didn’t get anywhere beyond the “bootstrap” phaze of the attack, I thought I’d take a look at w00t. I assumed, just due to the name, that it was the exploit that would yield root. Google didn’t really bring anything up, although I didn’t expect it would since “w00t” is such an elite term. You know… 3r33t and /|ll that (yes, I used to run a hacker bbs and know all the lingo).
So I figured, what the hell, I’ll run it (as user apache, on a separate test machine, also running Annvix). Yup. It’s a rootkit. Nice thing is, it didn’t work. =) strace doesn’t show anything particular; lots of segfaults. It’s definitely trying to exploit the kernel, however, as it gives this:
[-] Unable to determine kernel address: Operation not supported
uselib24 looks interesting as well:
[vdanen@build ~]$ ./uselib24
[+] SLAB cleanup
child 1 VMAs 401
child 2 VMAs 23
child 4 VMAs 03
[+] moved stack ffffc000, task_size=0x00000000, map_base=0xff800000
[+] vmalloc area 0x7d000000 - 0xf9f1b000
[-] FAILED: uselib (Function not implemented)
Killed
I have no idea what any of that means, but I’m suspecting another kernel issue. Oh well. [UPDATE: this kernel exploit is fairly old, the source can be found on packetstorm.]
Enough fun, time to clean this junk up.
Bottom line? Use a good fixed kernel (either a patched one from your vendor or the latest kernel.org kernel). I’m not liking the 2.6 kernel for servers yet, so I’m using 2.4.31 with the openwall kernel patches. Make sure your web apps are up to date; until the client fixes his wordpress install, I’ve set a apache deny clause to that directory so it can’t be used as an attack vector anymore. And firewalls that not only prevent incoming traffic but also outgoing are a good thing. Remember, default DENY and selectively ACCEPT. To DENY particular ports is pointless, if I had done that, the intruder would have gotten in (I have no reason to explicitly firewall port 1666 after all). But because I denied everything then opened selective ports (http, https, sshd, etc.) I got my butt covered.
Interesting stuff. Hopefully it’ll be another 3 years before something like this happens again. Of course, I’m going to have to keep on top of clients. And tighten the hatches a bit more to make it even harder, but I think overall, this one went pretty good and no real damage was done.