random($foo)

Archive for the ‘Tech’ Category

Newer Entries »
« Older Entries

Personal Data Storage 2011

Tuesday, February 22nd, 2011

The last time I spent much time on personal data storage was over 2 years ago. Since then, all my NASes and most of my disks have actually been themselves in storage.

While I’ve been nomadic, I’ve mostly been haphazardly filling up 2TB 3.5″ external disks and various 2.5″ portable disks. Lots of these are dupes however, so now that I’m a bit more settled, I’ll be trying to bring these in line.

To that end, I’m building a new storage filer, and potentially will build a second as a backup if it works well. I’ve decided to take the ZFS plunge so I’ll probably be running FreeNAS which has ZFS support and is USB stick friendly. Here’s what I picked up last night:

  • Patroit Xporter XT Boost 16GB Flash Drive – the 8GB is potentially faster, but price/MB is better on the 16GB. (the XT Rage is a bit faster than the Boost, but the casing can block other USB ports). There may be a 5th SATA port I can use with the included HD, but I’ll probably pull that and boot off of USB anyway, save some power.
  • HP ProLiant N36L MicroServer – this seems to be the best solution right now for a build-your-own NAS. It’s <$400, it's much faster than an Atom solution with about the same power consumption; supports 4 drives, sadly no hotswap. It’s actually about $50 cheaper at Newegg, but if you’re in CA, tax and shipping make it a wash w/ Amazon
  • 4 x WD 2TB Caviar Green WD20EARS drives – these drives are cool, quiet, and fast enough. I’ve been using them for years and haven’t had any problems. There was just a crazy $20/drive rebate that apparently I just missed, but jeez, $85 for 2TB? Just remind yourself how much that would have cost even 5 years ago.
  • 4GB 1333MHz DDR3 Non-ECC CL9 DIMM (Kit of 2) – HP thoughtfully includes 1GB of RAM with their MicroServer. Like the 160/250GB hard drive… it’s marginally useful. The server maxes out at 8GB, but 4GB for $44 is a pretty reasonable price. ZFS apparently really likes memory for cache, but lets get real. We’re not doing much mission critical work here.

I’ll post an update once everything’s up and running, including final kill-a-watt numbers.

Posted in Personal, Tech | 3 Comments »

Cloning Macs via the Command Line

Monday, February 21st, 2011

Recently, I’ve been doing a lot of cloning/imaging of machines. At a certain point, Carbon Copy Cloner and Disk Utility (or any GUI tool), doesn’t cut it. CCC of course started out as a wrapper for asr and there are guides for using asr for disk imaging online.

I’ve written a slick set of scripts that are streamlined for my purposes (automatically finding the mountpoints for sources and targets etc), but there’s probably some general stuff worth sharing…

Before cloning, I have a script that ‘preps’ the machine. This includes running periodic [daily|weekly|monthly] and clearing the caches:

sudo rm -rf /Library/Caches/*
sudo rm -rf /System/Library/Caches/*
sudo rm -rf /private/var/root/Library/Caches/*
sudo rm -rf ~/Library/Caches/*

Making the “master” image is pretty straightforward:

diskutil unmount "$volume"
time hdiutil create "$dmg" -format UDZO -nocrossdev -srcdevice "$device"
time asr imagescan --source "$dmg"

(I use a combination of time and overall timing recorders with date +%s epoch math to get proper performance numbers)

And cloning is even simpler:

time asr restore --source "$1" --target "$device" --erase --noprompt

Using --erase is essential to doing block-level writes and is a magnitude faster and more reliable.

Lastly, and what probably makes this writeup somewhat useful, I have some additional code that renames the hard drive and the machine name:

diskutil rename "$device" "$newname"
perl -p -i -e "s/$oldname/$newname/" "/Volumes/$newname/Library/Preferences/SystemConfiguration/preferences.plist"

The last step is preferable to using scutil because you can do it without rebooting, and it’ll already be renamed before there are any conflicts jumping on the network (which leads to the dreaded “(2)” problem).

With these scripts, I build a fully production ready clone in about 10 minutes flat.

Missing at this point:

  • Screensaver preferences don’t clone for some reason, I probably need to find a similar plist setting and rewrite that
  • I want a keyboard that just has a “T” key.
  • I probably should build a launchd script to run a “first bootup” script for anything else that needs to happen.

Posted in Tech | No Comments »

Making sure WordPress is up-to-date

Saturday, February 19th, 2011

For anyone that has to manage many (any?) WordPress instances, keeping it up to date can be a real PITA. (I don’t expect WP to ever have an SSH vs FTP based auto-update).

The best way to make life a little easier is to move over to using svn tagged versions. Then you can simply switch over the next time a security vulnerability is patched with a simple ‘svn’ switch.

Here’s some simple bash-ness that’ll help you know when to upgrade. In *theory* you could run this on cron and have it auto-update, but I’m assuming that if there’s a DB schema update required, you’ll need to watch over it, so probably not something you want to run on production without some verification:


installed_version=`svn info $yourwp | grep URL | cut -f 2 -d' '`
current_version=`/usr/bin/lynx --dump --nonumbers http://core.svn.wordpress.org/tags/ | tail -n 2 | head -n 1`

if [ installed_version != current_version ]; then
  # this is where you could get fancy and do an svn switch and update
  echo `date` | mail -s "WP is out of date!" yourmail@example.com
fi

Posted in Tech, Web | 2 Comments »

Checkvist is my Task List

Thursday, January 27th, 2011

Like most geeks, over the years I’ve gone through a lot of a lot of task management tools (most successfully, OmniOutliner and TVO), only to go back to using text files (which unfortunately, tended to accumulate on multiple computers at multiple locations, I still have a few megs of files tucked all over the place). Lately I’ve been using a lot of Evernote, and as one of the few things that synced flawlessly (although Dropbox has really changed the game there), I used that for about a year or so as my main task “manager”, and used FogBugz for development-related stuff.

After the latest round being really unhappy w/ my setup, I took another look around what kind of online/shared options were available and stumbled onto Checkvist early last year. Although there are others that do more (Toodledo, and of course, Remember the Milk), Checkvist was the one that stuck the best, mostly because of its simplicity and speed. Also, probably the biggest factor (and a big contributing factor to the speed aspect) is that it can be almost entirely keyboard driven. It’s a moded editor with tons of chained shortcut keys. You can see the appeal to a longtime vim user. It also has decent multi-user sharing baked in from the start, which is nice for small teams (although there are a few things missing that have been stumbling blocks for adoption by my co-workers).

That being said, despite a few things that originally seemed like dealbreakers (lack of mobile app, for one), over the past few months, Checkvist has managed to take over as my primarily task manager. I attribute it primarily to the parts that it gets right: keyboard driven UI, search, list switching, expansion among them, that are just unmatched in anything else I’ve tried.

Here, BTW is my ongoing Checkvist Evaluation/Improvement List:

Checkvist’s closest peers are probably Todoist, which has a few nice touches, but seems to be inferior to Checkvist in just about every way, and Workflowy, which is conceptually very interesting (dynamic scoping/zooming, keyboard everything, and while slicker, is too limited to be really useful for me (even less metadata, no sharing). Also, I suspect that the lack of modalness actually makes the keyboard nav a lot more complex than it otherwise would need to be…

Posted in MakeItBetter, Personal, Tech | 2 Comments »

My Mac Apps

Sunday, January 23rd, 2011

NOTE: I’m keeping a more up-to-date (and prettier version) of this list on Bagcheck.

One thing I’ve noticed having done a couple of reinstalls lately is that I’m using a lot less apps on my computer. I figure now (that I’m procrastinating) is as good a time as any other to write things down in case anyone finds it useful:

  • Google Chrome, Firefox 4 and Plainview
  • SIMBL and TerminalColours – pretty much the first thing I install, a must for seeting what you’re doing on a dark terminal background (I bump up green, blue, red, magenta and cyan). I’ve linked to the Network graph for TerminalColours, as the head seems to be ever-shifting
  • XCode
  • OpenTerminal – incredibly useful; I change the prefs to always open a new window and to open the selected folder
  • ClipMenu – the best clipboard history app I’ve found. I don’t use it’s snippets much, just increase the buffer to 50, increase the autosave frequency and show the last 10 inline
  • MacPorts – while others might have moved to Homebrew, MacPorts hasn’t failed me yet and I’ve invested the time to learning variants, upgrades, cleaning and activations. As long as it keeps working, I’ll keep using it. The first thing I’ll usually do is install python27 and python_select to python27. I finally learned my lesson not to use Apple’s built-ins. I also use MacPorts’ vim (+python27), curl (+ssl), and git-core.
  • Thunderbird – one day my Mail.app stopped working w/ GMail’s IMAP. I switching to browser-only, but that didn’t work well, especially w/ 5 GMail/GApps accounts. I’ve switched before, but after a year w/ Thunderbird 3, it looks like it’s taken for good. I have a few useful extensions, but by far the most useful is keyconfig, which I primarily to map the ‘y’ key to archive. I mostly use the widescreen view w/ Group By reverse chronological sorting in the unified inbox.

    • Alfred – I finally switched off of Quicksilver a few months ago. Now, there’s still some development going on, and it mostly works, but it felt like the right time to move to something a bit more vital. After trying Quick Search Box, I settled on Alfred mostly because I wanted a quick way to access individual preference panes. My launcher needs have always been pretty basic, I just wanted something that works and doesn’t crash much.

  • iStats Menu $ – For a long time I ran MenuMeters, which is great and does everything that I want. I mostly switched because iStats Menu has a nicer calendar than MagiCal, on par with the features I used from MenuCalendarClock (I can’t recommend the latter, as I registered, and shortly after a new version came out that required re-registering). Paying another $20 was a bit much for me.
  • Adium – the only feature I wish it had was cloud-based log syncing… hmm, I wonder if saving to Dropbox is a good option… I’ve use the Domo-kun notification icon for years, although the Totoro is pretty tempting. I make a lot of customizations to my Adium, but by far the most importan are deleting Growl notifications, turning off sounds, disabling the menu icon, and hiding messages and lists when backgrounded – basically, anything that would actually interrupt my work (I have my dock hidden by default, otherwise I would disable that stuff as well).
  • VLC – must have for MKVs and the like (I’ve used Perian in the past, just couldn’t be bothered to reinstall it)
  • Cog – I loathe iTunes, and Cog is the best lightweight music player I’ve found. The official release hasn’t been updated in a while. I use the latest nightly without problems. Vox is a slicker alternative, but doesn’t have playlist management or scrobbling (it does!). (Cog does via the Last.fm Scrobbler)
  • DTerm – I’ve yet to really start using it, but the copy results to clipboard seems really nice.
  • Alarm Clock – I found this a few years ago, and it does everything I want/need in an alarm clock.
  • Dropbox $ – I pay for the Pro account and use it everywhere. Here’s an interesting Quora question on why Dropbox is so much more popular than competitors. The simple answer is that it just works.
  • Carbon Copy Cloner – Its read-only image writing seems to work much better than SuperDuper‘s for some reason. With it, I can image a machine (11GB uncompressed) in about 5 minutes.
  • TeamViewer
  • 1password $ – this works I think as well as can be expected. LastPass is probably its only real competition. One slick thing is that if you’re syncing with Dropbox, the 1password’s license file will be automatically synced.
  • Evernote $ – It’s not perfect, but it does that one thing well that no one else did (sync!) With 2.0 (folders!), I find myself less angry at it now at least.
  • Gabble – a far better Yammer app than the AIR one
  • Sequel Pro – as much as I live in mysqlclient, Sequel Pro does some really convenient things (duplicating rows for example) and is a lot faster than MySQL Workbench.
  • TinkerTool – I only use this app to move the Dock (I run mine hidden on the top left side w/ only running apps), although I include it in the list because it’s does other neat stuff as well.

OK, not the shortest list, but compared to the number of apps I had installed before…

Posted in Personal, Tech | 5 Comments »

Looking Back, Looking Forward

Friday, December 31st, 2010

Charlie Stross posted an interesting essay today, Reasons To Be Cheerful recapping some of the great things that have happened in the world over the past decade, primarily in the developing world. A great read, and honestly inspiring/heartwarming for the disheartened humanists. It’s easy to get overly cynical about it all. This is a good antidote.

That being said, I don’t think Charlie goes quite far enough. The essay starts framed by the thesis that in the world, things haven’t much improved, and the besides a few specific counterpoints about disease and the general march of technology, it feels like he gives up on really repudiating that thesis… for the developed world. And it’s easy to see why. In terms of general socio-economic trends, it’s hard to be all that positive. Things are downright unsettling heading towards dystopian. However, there’s at least one aspect, the very medium where we are commenting on that is worth, uh, commenting on.

Yes, the interwebbytubes, as Stross puts it, is quite a different place than it was at the beginning of the millennium. We are looking at a 2X adoption growth in developed nations (from plurality to supermajority, if not ubiquity). Worldwide, 2 billion people are now online. Beyond the quantitative changes, the qualitative changes are even more intriguing. In 2000 there was no Web 2.0. Blogging was in its infancy. Most of the things we take for granted online today were not invented yet. Among them: Wikipedia (2001), Facebook (2004), Google Maps (2005), Twitter (2006). I list these in particular because I don’t think there’s a day that goes by where I don’t use these particular services, but I’m sure that others have their own lists. Lest you think that this was a singular period of growth, I’ll throw in that the iPhone (2007) and iPad (2010) have kicked us into another era of hyper-growth that will be just as (if not more) life-changing.

We’re just starting to see what happens when the Internet starts engaging with us in a location/context aware fashion. We’re also starting to see what happens when Internet-style/scale dynamics are applied outside traditional consumer Internet contexts (e.g. Obama Campaign, 2008). In a historical scale, we’re still at the very beginning stages of figuring out what it means to live in a digital, massively inter-networked world, and similarly just starting to get a handle how that will change society (attention, communications and collaboration in particular).

All that’s a really long way of saying… well, there’s a pretty dang bright spot in the developed world too. One that has the potential of being turned into the shovel we need to dig ourselves out. So, here’s looking to the future. Happy New Year.

Posted in MakeItBetter, Personal, Politics, Ramble, Tech, Web | No Comments »

Wikileaks, Net Neutrality, Architectures of Participation

Friday, December 24th, 2010

This post is mostly a placeholder/notes for further thinking I’ve yet to do about a few related threads that seem connected this past week. Before, but particularly since my experience working on the 2008 Obama campaign, I’ve been thinking about the most potentially transformative aspects of the technologies that we deployed: specifically, deploying methods and means for self-directed organization and participation.

In the meantime, the things that some things that have caught my attention.

In regards to the capitulation of Net Neutrality, this thread on building a alternative mesh network. I wonder if it’ll come to that?

On Gitmo and normalization of indefinite detention, davidasposted’s sobering analysis of the situation.

And of course, there is Bruce Sterling’s Wikileaks missive – melodramatic, oversweeping, but truly compelling, and a must read (counterpoint).

Also, Julian Assange’s impressively articulate recent interviews, and more information on Bradley Manning’s continued mistreatment.

Posted in MakeItBetter, Politics, Tech | No Comments »

Gawker Passwords, etc.

Monday, December 13th, 2010

I have work deadlines, so I haven’t been able to been able to write a well constructed post about this, however, a few things:

  • To check if you had a Gawker account (there are 1.25M of them, so you might have one even if you didn’t realize it) I recommend: http://gawkercheck.com/. Note: even if your password wasn’t unhashed, consider it compromised. These passwords are encrypted with DES crypt, which is not adequate to stop attackers. The keyspace is too small. For more info on DES (and probably the best post-mortem so far), see this Forbes blog post.
  • This is as good a time as any to manage your passwords properly. A lot of people (including me) are using 1Password. It’s currently available as part of the MacUpdate December 2010 Software Bundle. LastPass also looks like a good solution and is free ($12/yr for mobile support). PwdHash and KeePass are also options.
  • According to the FAQ, Gawker claims to be sending emails eventually (and some people are doing so as well now). What I did last night, and maybe a good thing to do for your friends if you are an uber-geek is to go through your friends list and grep through the torrent database and them personally know if their account has been compromised, especially if the password has been unhashed.
  • Oh, lastly, if you’re a geek w/ your hash and want to check on whether it’s a reused password or not, you can pretty easily fire up a python shell and see if it matches: 
    password = 'your_password'
hash = 'your_hash'
salt = hash[0:2]
import crypt
crypt.crypt(password, salt)

    If you’re not sure though, audit your passwords anyway when you have a spare hour or two. You’ll feel better, trust me.

Posted in Personal, Tech, Uncategorized, Web | 4 Comments »

Learning New Things

Saturday, December 11th, 2010

Today was an average afternoon – taking way too long to accomplish a seemingly trivial task, but looking up and learning a bunch of new things along the way. It seems there should be a better/easier (almost automatic, transparent) way to track the sources (links/pages), process (things tried) and results (code fragments)….

The basic goal in this case was to automate some execution of some javascript on a page. Because execution of the script caused a page load, it wasn’t a matter of writing the calls into the console. The faster and easier way would have been to write a Greasemonkey or Chrome Extension script (because there were timing issues, the script would have to write a time-based state file on actions), however, I figured I would try to see what kind of options were available with a control-script oriented model, as having that handy would be more generally useful in the future (more and more, straight mechanize is less useful as more JS proliferates).

Before getting started, I had to strip out just the lines that I wanted. I always forget how to do it, but that was a simple vim command lookup.

I looked at Selenium and Selenium RC, which probably would have worked fine (but that I didn’t use because I didn’t want to install extensions and the RC docs weren’t directly linked, but would have probably saved me time in the end).

Instead I decided to try out Watir (cross-platform, cross-browser, and my Ruby is rusty so this was a good excuse). I started out with SafariWatir, however, after a bit of poking, came up with a dead end on executing JavaScript. There’s a scripter object, but even after getting access to it via monkey-patching (did I mention my Ruby-fu sucks?), I was still getting errors and there wasn’t much help In general.

Instead of slogging through a potentially losing battle, I decided to jump ship to FireWatir. FireWatir uses JSSh, which communicates directly via JavaScript to Firefox, so it seemed like it might be a surer thing. My Firefox profiles were corrupted from my last system transfer, so there was a bit of messing with the profile folder until I gave up and started a new, but after that it seemed like I was home free.

Except, that when running js_eval, it turns out the scope that JSSh puts you in, isn’t in the document DOM, but rather the browser’s XUL DOM. For whatever reason, I couldn’t find a reference for even with the direct object type refereces (i.e. getWindows and getBrowser return ChromeWindow objects, which just don’t seem to have docs. Introspection via domDump() or inspect() just returned a huge amount of stuff to go through). Luckily, while searching, one of the results that turned up was a StackOverflow question on firewatir + jQuery which answered the question – ChromeWindow.content gets you into the HTMLDocument DOM. I’m a bit mystified why this isn’t in the firewatir or JSSh docs, as this seems to be one of the most common things that people would want to do, but well, that is the life of the developer…

Posted in Personal, Tech, Web | 2 Comments »

The iPad While Travelling

Saturday, August 28th, 2010

I’m actually about to head out on another extended trip (3 weeks in Taiwan, 2 weeks in Japan), but I thought I’d take a few minutes to write up how the iPad was on my first long trip (to Australia, Fiji) while it’s still somewhat fresh.

It was definitely convenient on the long flights to and from Sydney (about 14 hours each way) and I barely broke out my laptop on both those flights – while the 16GB is fine for most uses, when you’re putting on HD movies that are 4-8GB in size… well, there was definitely some laptop swapping (definitely cursing iTunes and wishing the iPad had some way to read external storage directly). As an aside, I highly recommend MKVtools, which will do intelligent container shifting for many devices. For the iPad, you usually won’t need to re-encode the x264 video track, just the audio (usually AC3 or DTS) into AAC.  It’s a one-click operation for the iPad, and is much faster than transcoding (taking about 10 minutes instead of 10 hours for a movie). It’s free to use one-by-one, but totally worth the $5 to unlock queueing of encodes.

I flew economy on VAustralia, which while not exactly cramped, was certainly more comfortable with the iPad than a laptop open.  In economy, the seats also had USB power, which was also convenient, although if you’re fully charged up, actually unnecessary.

I ended up carrying my iPad just about everywhere, as I picked up a free data SIM (see my previous mobile data writeup) and had a perfectly sized bag. (only $20, and with just the right amount of extra pockets, super recommended.) The nice thing is that unlike my trusty Chrome bag when loaded with my Macbook, there wasn’t really a moment where I felt dragged down, even going up and down Sydney’s biggest hills. (I know, right? Who knew? Sydney is super hilly!)

While I had my laptop nearby most of the time, I decided to head to Fiji with just my iPad to see how well that would work out.  While that wasn’t too bad, it was mostly due to my being mostly disconnected (there was very little wifi and I barely got mobile data working at about $2/MB with a Vodafone Fiji SIM). I found that there were a couple times where I wanted USB charging and transfers, mostly for my camera, but also when I rented a car and it had an SD slot in the stereo. (!)  The iPad did, however work with the prepaid internet access at the airport despite the system’s warning that browser popups needed to be supported.

It was an interesting experiment, but even not doing any work, it seems that the iPad isn’t quite there yet as the only thing you carry traveling.  Not if you plan on taking photos or want to move any files around.

So that’s it for my iPad report.  The Toshiba AC100 is out in Japan, Taiwan, and in Europe, so that’ll probably be the next report (maybe, unless Apple releases a dead sexy 11″ Macbook Air, or I’m magically swayed by the Vaio X; I’ve written at length, however, about why I’m particularly interested in smartbooks, even over other ultraportable options).

Posted in Personal, Tech, Travel | 3 Comments »

Newer Entries »
« Older Entries
« Older Entries
Newer Entries »