Tonight I was working on on-the-fly tarring of files, and in the spirit of not-reinventing-the-wheel, a quick search turned up a just written article, Creating ZIP and TAR archives on the fly with PHP. Great!

Except that when you think about it, TAR (that stands for Tape ARchiver) is built for streaming. It just doesn’t make any sense to create a temp file and then send it – that seems like a waste (unless you were streaming the same file many, many times or using a separate set of servers for sending static files). If you’re going to be sending a file with the same Apache processes, here’s the right way (well, my right way, YMMV) of on-the-fly creation:

// Unlimited execution, as long as it takes to DL
set_time_limit(0);

// For proper browser handling
header('Content-type: application/x-tar');
header('Content-Disposition: attachment; filename="' . $filename . '.tar"');

// Make it safe
$filename = escapeshellarg($filename);

// the C argument is so that doesn't get included in your tarball extraction
$cmd = "tar cC /path/to/tar/from $filename";

// teh magic
$fh = popen($cmd, 'r');
while (!feof($fh)) {
  print fread($fh, 8192);
}
pclose($fh);

Upcoming.org was originally launched on PHP4 and I believe, MySQL 3.23. As you might imagine, internationalization (i18n) wasn’t at the top of the priority list. As time has gone on, more and more international users have started to use Upcoming. With global geocoding and time zones off our plate, it was time to tackle character encoding.

Now, everyone knows that that i18n is easy: just use UTF-8 across the board and you’re done. The reality of course, is more complicated. While I don’t think there’s any disagreement that Unicode is a Good Thingâ„¢, even a decade and a half in, support is still wildly uneven in different environments and programming languages. No doubt these notes will be out of date soon, but I’m writing this down so I know where I can pick up next time.

Development Environment

The first thing to do is to make sure that you’re working in a UTF-8 aware environment. Browsers are good with encodings (a little too good with character-set autodetection, but you can force them into whatever display you want, more later). OS X is Unicode friendly and Terminal.app is a good place to start. Linux is also pretty good, you can check what you’re locale settins are w/ “locale” – RHEL defaults to en_US.UTF-8, so display in xterms actually work out of the box (although the Unicode fonts seem to be missing lots of codepoints).

Beyond your locale, you’ll also want to make sure that your pager can display UTF-8. You can export LESSCHARSET=utf-8 into your bashrc.

Lastly, and for me, the most important tool is Vim. Version 7 has improved unicode support, but for me, v6.1 worked fine in binary mode. Being able to edit a file (vim -b), move over to the character, and type g8 to get the hex code points was the easiest way for me to verify transcoding/character set issues.

MySQL

We’re now using MySQL 4.1.x, the later builds which have pretty good character support. There are still outstanding collation issues, and things like UTF-8 corruption in InnoDB for <4.1.16 and character_set_name problems <4.1.19 are troubling, but overall, once you the bajillion character_set variables to "utf8", thing seem to work alright. Since we remained in latin one when we went from 4.0 to 4.1, we avoided some big headaches, especially since we actually were storing many different character encodings (MySQL 4.0 and before simply treated values as binaries, so this wasn’t a problem). A couple of alters later and we’re now in utf8/utf8_general_ci with a mixed-bag of binary data (untouched by the alters).

Python

Of the languages that we use, Python has the best unicode support (just watch out for the wacky print behavior; another howto). That and Mark Pilgrim’s chardet package made it a no-brainer to do character set conversions in Python. Unfortunately, I ran into some hairiness with trying to get MySQLdb to write UTF-8. That eventually got solved (if you’re running a setup w/ broken set_character_set() then force it through the my.cnf), I worked around that problem by sending the UTF-8 through the mysqlclient.

Perl

Perl has had, um, issues with Unicode. 5.8 apparently fixes lots of things – too bad I’m running 5.6.1. Still, I didn’t have too many problems with it. The localized scope utf8 pragmas are sort of weird, but seem to work as advertised (so glad I coded all my string handling in a modular manner). The one problem I had with DBI not returning UTF-8 was solved with a quick search – interestingly, SET NAMES worked perfectly for DBI even when it did bupkiss for MySQLdb.

PHP

PHP had potentially the worst Unicode handling, but PHP5 improves things a lot, with native UTF-8 support (just remember to set it as the default character set). Since the stuff I was working on didn’t require string manipluations at all, I completely lucked out here. The only thing I needed to fix was a stray utf8_encode() call that was crushing the UTF-8.

Summary

So, ironically, I ended up having the most problems with the language with the most complete Unicode support. This whole deal was quite the PITA, but on the bright side, it’s done, and the everything seems to have squeaked by with at least a passable level of workingness. (I’m not sure this would have worked so “smoothly” even as late as last year.)

A couple years back The Futureheads released an album to great acclaim. While I liked some of the tracks well enough, and I could recognize it as being undeniably good, it just never really caught my ear. I’m happy to say that I’m digging the new album much more.

• The Futureheads – Thursday
  

Recently with the impending US release, I finally got the Nintendo DS bug. I don’t know what tipped it over for me (probably SylphIRC), but it looks like there’s some momentum at the office. Ed, Kevin, and Eric are all getting NDSLs.

To be able to run homebrews, emulators, backups and the like, you’ll need two things. First you need a passkey to boot into DS mode. The latest and greatest is the Max Media Launcher, which replaces the older bulky passkeys with a DS cart sized (but non-reprogrammable) replacement. (You can subsequently go through a flashing process (link to FlashMe to bypass the need for a key, but it seems like sort of a bad idea for the DS Lite at the moment). In any case, I bought mine for $26 shipped from the UK.

UPDATE: It looks like the MML is buggy with sleep mode. To get around this you can swap in an NDS cart after boot up. Alternatively, the Passkey3 and Super Key are both similar NDS cartridge sized launchers that are supposed to be coming out soon.

The second thing you’ll need is a memory cartridge / player. The best at the moment seems to be the ones made by M3 (100% compatibility, skinnable interface, media playing, lots of emulators built in). The Mini-SD version is very slightly shorter than the SD version, but SD cards are cheaper and come in bigger sizes (and since I have SD and mini-SD cards already, it makes more sense to have something that can take both). While the newest SD SuperCard is smaller (GBA-cart sized) I figure I wouldn’t be bothered too much since GBA carts protrude from the DS Lite already anyway. What’s another centimeter or two.

Here are some links of the things I’m most excited about:

• ScummVM DS – that’s right, Monkey Island, DOTT, and Sam and Max on your DS. Sweet!
• DS Linux – need I say more?
• Homebrew Nintendo DS Development – these tutorials look fun (more dev links)
• SylphIRC – IRC on the DS; useless? maybe, but still damn cool
• Homebrew Apps
• 4 color rebellion » State of the Homebrew – May 2006

Lately, screencasting has been gaining steam on my internal radar (Gordon’s working on a very cool side-project right now). I just made some since it seemed to be the easiest way to show my mom how to do some basic computing tasks (It took about 20m to make 3 short ones).

I wonder if there’s a collaborative screencasting/tutorial site? It could actually just overlay on top of YouTube for storage, etc, with some additional metadata/filtering attached…

Chris introduced me to Cloud Cult, whose latest album (released last year), is amazing.

Here’s one of the songs that I’m especially hooked on:

• Cloud Cult – What Comes at the End
  

Paul Robertson‘s epic video-game inspired sprite-based animation is amazing.

Last year, I set up an “ISP-level” mail system for myself. Well, it seemed to me like I had some pretty common requirements: being able to handle mail for multiple domains without having to create UNIX accounts, and secure logins. Despite “email” being 30 years old, however, it turned out that there was no simple package that I could apt-get to install this. Instead, I ended up almagating various tutorials/packages for my Postifx w/ SASL + Courier IMAP w/ SSL + MySQL configuration.

One piece I wasn’t able to set up, however, was virtual user mail filtering. I banged my head until finding out there was just no way to do procmail delivery w/o an associated UNIX home folder. After which, I moved on to looking at how maildrop might work, but after many fruitless hours I gave up.

Recently, I decided it was time to get it working, no matter how long it took, and after a few hours of hammering away, I finally have gotten it done.

• If you have a setup like mine setup except for maildrop, this is the clearest tutorial.
• If you’re running Debian, you’re going to need to compile your own Maildrop w/ MySQL support enabled. Note, that Maildrop 1.7.0 is the last version that has built-in MySQL support; after that version, Maildrop switches to using Courier authlib for connecting. I have the authlib stuff set up for my IMAP/POP, but I couldn’t get Maildrop to connect properly!
• test your maildrop setup separately as root:

  maildrop -V 10 -d user@domain.tld < test.eml

• If you get an maildrop: signal 0x0B error, use strace to see what's going on. Maildrop segfaults on bad configs. Also, be sure to turn on MySQL logging to see what query maildrop is actually performing