random($foo)

You’ll have to excuse the lack of posting or other productivity lately… I’ve been programming Java.

*buh-dum-dum*

This is for work, as I’m not usually so masochistic. Right now we’re crunching to push out a uPortal based student portal (to replace the stopgap I wrote).

Now before I rant, there are some good things about Java. Mainly that Eclipse is a pretty stupendous IDE. Having advanced code-insight, library exploration, and a real live debugger are great things (oh, and try out that refactoring functionality – really slick). But then actually writing Java, there are the things that in theory make it good, but in practice don’t.

Oh, it starts innocently enough. Some patterns, interfaces, oh, and factories for everything, and then some property files… my God, the property files… And then all of a sudden, you’re on the other side of the line. You’ve just spent 20 minutes boxing/unboxing primitives, creating new Long objects just so you can increment (all those convenient data structures that require Objects, you know), or you’ve spent another half an hour chasing down wrappers and managers and inherited classes when all you want is to write out a simple parameter (ugh, I don’t even want to start bitching about XSLT right now)… Is this supposed to be easier?

Anyway, I can see who David is selling Rails to when he talks about build[ing] real-world applications in less lines of code than other frameworks spend setting up their XML configuration files. (On the other hand, I’ve been running into code scaling problems of my own w/ PHP, especially wrt handling complex front-end code. Obviously I don’t have the answers, but I’ve started playing w/ Rails, and I’m not sure that it’s the answer, especially on the JS integration end).

Well, that wasn’t much of a rant. Too tired to care that much. Some links of some tools I’ve been using lately:

• TestXSLT – I wouldn’t go as far as saying it makes playing around and learning XSLT fun, but it’s definitely the best OS X XSLT tool I’ve found so far
• skEdit – I’ve been trying this out, decent but not really astounding. I’d rather have a Mac port of TopStyle
• DbVisualizer – a tad slow, but good for digging around
• iSQL-Viewer – another useful JDBC DB tool

Yesterday Tom posted three articles on a really rough proposal for an Apple Media Hub. I’m not up for writing anything long (or coherent) right now, so some random thoughts:

• I see a much more direct link between a media center (playing/organizing digital media) both to acquisition and organizing of personal data and cataloguing of physical media
  • automatic population of a Delicious Library type application based on media usage (might as well catalogue as you play, right?)
  • As the media hub will likely be many people’s first file server, easy and automatic syncing and remote backups would be something I think would be pretty obvious, especially if it’s acting as a gateway as well (automatic encrypted backups of my Quicken files would rock)
• Watching the finger contortions a non-techy friend put to do what he wanted w/ his PVR and HDTV setup really opened my eyes 1) about how people have taken to the power these new appliances have given them, but also 2) how the interfaces suck
  • Macros, etc: so, nothing that can’t be done with super high-end remote controls, but why not go one up and use a simplified OSD interface, something Automator like
  • Scheduling: and not just limited to media applications — I think agree that home automation is definitely a logical next step

Apple introduced its Mac mini today. It looks great. Add a breakout box w/ a nice VFD (I’d like 2 hdtv tuners, component in/out, toslink, spdif coax, and 6+ channel analog out) and you’d have a perfect base for a true digital hub. Personally, I’m hoping that sooner, rather than later, someone will release a platform w/ the building blocks for betting tapping into the opportunities that convergence could provide.

Addendum: I just spotted an interesting post about the mini and automobile computing – this is a perfect illustration of the kind of potential I see w/ convergence. What’s important to note is that general computing doesn’t go away, but rather gets infused into embedded applications. What used to require embedded toil and custom hardware instead moves towards scripting or even higher level development. Tinkertoys ready for the pro-ams.

You won’t find me writing about sports very often, but holy crap, our team just completely kicked the crap out of the Sooners.

Oh, also, it gave us a chance to put up something new on the home page.

One of the most exciting things for me about the new OmniOutliner 3 is its auto-save capability. This is a feature that is configurable in the preferences (defaults to saving every 5 minutes).

Unfortunately, I just discovered that rather than backing up into a separate folder, auto-saves are actually committed directly to the outline files. This means that files that haven’t been saved at all (say the 20 untitled outlines I had open) won’t be around after your power mishap.

FYI.

First of all, to help with Tsunami relief, Google’s page is a good start. As is Amazon’s Red Cross Diaster Relief page ($3.8M and counting), where I donated. Apple.com also has direct links to relief/support agencies.

I threw in two machines tonight to help out Andy w/ his Tsunami videos that he’s been hosting. There’s apparently a ginormous demand, mostly coming from Google searches (#2 for tsunami video).

On my EV1 server, I’m averaging 85Mbps+ on the 100Mbps interface, and have just upped MaxClients and ServerLimit to 600 (after hitting 500). At this rate (36GB/hr) I’ll have to take it off the rotation tomorrow. Same with my SM machine, which is averaging about 21GB/hr. Between the machines on the round robin, I’m guessing that conservatively, we’re averaging at least 100GB/hr.

What’s interesting is that this dwarfs the BitTorrent traffic of the videos being served elsewhere. From my earlier Crossfire and Internets Vets experience, I’ve found this ratio to be between about 50:1 to 100:1 when both the torrents and direct links are given equal visual standing.

This isn’t to say that CacheLogic’s traffic numbers are inaccurate (they say BT accounts for 35% of all Internet traffic). I believe it. But the numbers I see from my own experience lead me to believe that the majority of that traffic is still being consumed by a relative minority of users (centered around predictable types of content), and that most people still tend toward direct downloads when available, regardless of performance.

Blog Torrent and Prodigem look like steps in the right direction for simplifying tracking, but I believe the fundamental bottleneck still lies in user adoption. Unfortunately, I don’t think this problem can be solved in a significant fashion until torrents are made transparent either by being integrated/bundled with browsers or available as ActiveX and XPI plugins, but we’ll see. In the near future, I should be able to have a much larger dataset to work against.

[Update: Hosting these tsunami videos was interesting. For the day that I threw in, at no point was transfer below 150Mbps (about 1TB of transfer). The bandwidth available was probably over 0.5Gbps and it was completely saturated. After Andy moved the videos over to Archive.org, the demand took them offline. When you have that much demand, it becomes pretty obvious how to get people to use BitTorrent – make it the only option available.]

There doesn’t seem to be a good detailed HOW-TO, guide, or tutorial compiled on the subject of ReiserFS recovery from partial drive failure, so I figured I’d give it an ol’ writeup, seeing as I have some time to kill (about 10M blocks left to be corrected).

Before I start out, I should mention that good backup procedures are a really good idea. If you’re actually using this advice, you probably realize this post facto. All hard drives fail. No two ways about it, so if you care about your data, have good backup procedures. I heartily recommend rdiff-backup, which not only does rsync-based transfers, but also, as the name implies, keeps increments. For Mac, I recommend psync to a sparse-image DMG for preserving resource forks. Carbon Copy Cloner does some pretty useful automation (in addition to ghosting great). Backupninja looks like a good rdiff-backup script, but I haven’t used it personally. Now, if only I could find a good transparent compressed volume format…

All that being said, sometimes you can get sloppy or lazy. In my case, I was building a new RAID system when a drive on my old file server crapped out… (no need for condolensces, this happened a long time ago, I just got around to doing recovery now)

The first thing you want to do if you notice block errors is to copy everything off. You might want to check smartctl and hddtemp to see if there’s anything really horrible going on before you do that. If your data is really important, shut down and send it to the professionals immediately. Ontrack is the most high profile, but there are probably others that can help you. Otherwise, try to copy what you can elsewhere.

If your copies were sucessful, at this point you can probably chuck your drive, recovery won’t be worth your time. If you can’t mount, or your copy wasn’t successful, then you’ll want two things: a recovery drive with enough space to hold an image of your drive, your drive unmounted. In general I’m of the opinion that it’s better not to spindown — it might not come back up (if you have a drive that won’t spin back up, usually accompanied by a click-whir, and sending it for data recovery is out of the option, I recommend giving freezing a try, it’s worked for me in the past), but it’s sort of hard to say what’s more of a risk. If it appears to just be bad sectors, I’d say turn it off until you’re ready to proceed if you don’t have a recovery drive.

If the drive affected is your primary partition, you may need to reboot with a boot CD. RIP is good (the advantage is it has dd_rhelp preinstalled), but if you have more esoteric hardware (say a separate HighPoint controller) you may want to go directly to Knoppix.

Once you have your old drive [hdbad] and new drive [hdgood], you’ll want to run dd_rescue, or probably better, the dd_rescue helper script dd_rhelp:

dd_rescue -A -v /dev/[hdbad] /dev/hd[good]

This will replicate the current drive onto the new drive (you can dd_rescue to a disk image file instead and mount -o as loopback if you’d rather). From then this point on we’ll try our recovery on this duplicate copy. Note that this differs from what others have said. They recommend backing up (w/ dd) and then recovering on the bad drive. Please read what they have to say, but IMO that’s a bad idea:

1. A plain dd won’t handle bad blocks correctly, so the backup probably will be messed up
2. Working on the bad drive will probably make things degrade more, very quickly. Running badblocks -b 4096 actually led to creating more bad blocks (including my superblock, sigh)
3. Even after feeding the bad blocks file into reiserfsck, it’ll still barf on bad hardware (again, speaking from firsthand experience)

So, I say, work on your dd_rescue’d image directly. This “backup” won’t do you any good if you can’t fix it anyway. Once you’ve moved your image onto a good drive, you can run a check

reiserfsck --check /dev/[hdgood]

It’ll tell you whether you’ve escaped unscathed or whether you’ll need to proceed through the levels of being ‘fscked’: --fix-fixable, --rebuild-tree, --rebuild-sb. reiserfsck is adequately descriptive in informing you on the level of your woes.

Note, I had more success w/ reiserfsck recovery running knoppix26 than the regular 2.4 kernel. If you run into an impasse w/ one you might want to try the other.

It’ll be a couple more hours before I know how this turns out.

Resources:

• TiVo Community Forum: How to revive a fried drive with dd_rescue – a good rundown on drive recovery w/ dd_rescue
• TiVo Community Forum: dd_rescue copy time – tests results and optimum way to salvage a bad drive – tests on increasing speed of transfer (changing block limits to cache size); also on saving to an image file
• Ultimate Boot CD – a crapload of (potentially) useful recovery tools. Very neat, even if they didn’t come in handy this time
• Bad block handling in ReiserFS – reiserfsprogs-3.6.12-pre1+ supports bad blocks, but, IMO still not that well. This is a decent page to read for reference, but not very helpful for recovery. Feeding a bad blocks list didn’t help in recovery
• Disk Repair and Recovery for Fun and Profit – an amusing link I came across during searching. It’s on manual floppy disk recovery and was written Dec 14, 2004. There are people still using floppy disks in 2004?

Ahh, cleared out 4000 messages. Just thought I’d enjoy it while it lasted.

Inbox Stats
---
Inbox: (0)/24 messages
Size:  4.8 MiB

Today's Incoming Mail
---
Today's Inbox:   
Today's Spam:    43
Today's Total:   83
Percent spam:    51.8%

Totals
---
Avg Mail/day:    179
Avg Spam/day:    65
Percent spam:    36.4%

Missed Spam:     58
False Positives: 53
CRM114 accuracy: 99.643%

Wow, this Metafilter thread is just plain depressing [the thread discussion was mostly about attacking each other, although the original post link was on a depression study: think of 3 snarky comments, then read on]. Granted I haven’t been reading mefi regularly recently, but it seems the sense of community and (well, sometimes) discourse has been at this point completely displaced by asshattery. Just one of the complete WTF moments:

db: Sorry, I posted before reading your apology. But on the other
hand, this is mefi: if you don’t want to run with the big dogs you can
always stay on the porch.

posted by Turtles all the way down at 7:12 PM
PST

Who is this guy I’ve never seen before talking about big dogs? Oh, joined this year. Actually, pretty much everyone in this mess joined this year. Honorary distinction for 327.ca, who should probably know better.

“Blaming it on the newbs” is a broad stroke, I agree. Metafilter will survive, and reach some sort of equilibrium as new members are enculturated, like it has done every time membership has been opened, but I think that it may be reaching a point where some additional features might help with scaling issues:

• Killfile (w/ public stats, details) see also below
• Extension of contacts system to allow flagging/highlighting/deprecating of threads and comments
• Overlay w/ user information/stats in each thread
• Some sort of per-post (reflected to user) rating. This should be kept a rubric on contribution to conversation (could be multi-axis if there’s a good interface)
• Community censure

I’ve listed those suggestions from what I think is easiest to hardest. Actually, the last one might not be the hardest to implement from a technical perspective, but I think has the most implications and really needs to be thought about in a larger context of what Metafilter’s goals are.

[Just some disclosure: I started reading Mefi in 1999, but stopped for a long while, then got an account when I picked up Cold Fusion. (worst. web language. ever) I went to a mefi meetup once but it was weird.]

Update: Matt links to this on MeTa (I think stinkycheese hits on a pretty good point). Also, to elaborate with some context on my technological suggestions (for what is essentially a social problem), the point isn’t that people can’t or shouldn’t disagree, but to help encourage an environment of productive discourse. I think that civility and empathy should be at least two minimal requirements. That being said, I don’t speak for anyone else in the community, who collectively will ultimately decide its character.

[not quite finalized, but close]

It’s about that time of year, so I put together a my top-10 albums released in 2004. Listed from discovery in, how else, reverse chronological order:

• Bettie Serveert – Attagirl
• The Arcade Fire – Funeral
• Erlend Øye – DJ Kicks
• The Kleptones – A Night At The Hip-Hopera
• Julie Doiron – Goodnight Nobody
• The Thermals – Fuckin A
• Ted Leo and the Pharmacists – Shake the Sheets
• Eternal Sunshine of the Spotless Mind Soundtrack
• Mirah – C’mon Miracle
• Iron & Wine – Our Endless Numbered Days

It was definitely a toughie to cut down to 10. Here’s another 10 that didn’t make the final cut. These were all great albums and managed to keep my attention this year (ranked by closeness to being in top 10):

• Ratatat – Ratatat
• Pinback – Summer In Abaddon
• Delgados – Universal Audio
• The Good Life – Album of the Year
• The Album Leaf – In a Safe Place
• Flotation Toy Warning – Bluffers Guide To the Flight Deck
• Blue-eyed Son – West of Lincoln
• Mellowdrone – Go Get’Em Tiger
• Grand National – Kicking The National Habit
• Sondre Lerche – Two Way Monologue

Here are some albums that I missed in 2003 that I caught this year:

• Guster – Keep It Together
• Some By Sea – Get Off The Ground If You’re Scared
• Ted Leo and the Pharmacists – Hearts of Oak
• The Stills – Logic Will Break Your Heart

For 2005, so far I’ve been listening to the new Low and M83 albums.

2004 marked the year that I took my music exploration to the next level. Lots of random clicking in a.b.s.m.indie. It’d probalby make more sense to make a top ‘new to me’ list. Maybe next year I’ll have a good automated solution to track what I’m listening to when (sorry Audioscrobbler, you’re just not cutting it when it comes to returning useful metrics/data).