Old Code: pystatsd-flickr

I was digging out some old code and realized that it might be useful for some, so threw it on github. Nothing special, just a lightweight python port of some of the old Flickr statsd that I wrote… almost two years ago. (time flies!)

If you’re looking to run something fancier in production, it looks like Etsy has some great stuff going on (see also).

Anyway, if you just want a quick graph and/or don’t want to setup Graphite, the code I’ve put up should get you generating an RRD pretty quickly.

Personal Data Storage 2011

The last time I spent much time on personal data storage was over 2 years ago. Since then, all my NASes and most of my disks have actually been themselves in storage.

While I’ve been nomadic, I’ve mostly been haphazardly filling up 2TB 3.5″ external disks and various 2.5″ portable disks. Lots of these are dupes however, so now that I’m a bit more settled, I’ll be trying to bring these in line.

To that end, I’m building a new storage filer, and potentially will build a second as a backup if it works well. I’ve decided to take the ZFS plunge so I’ll probably be running FreeNAS which has ZFS support and is USB stick friendly. Here’s what I picked up last night:

  • Patroit Xporter XT Boost 16GB Flash Drive – the 8GB is potentially faster, but price/MB is better on the 16GB. (the XT Rage is a bit faster than the Boost, but the casing can block other USB ports). There may be a 5th SATA port I can use with the included HD, but I’ll probably pull that and boot off of USB anyway, save some power.
  • HP ProLiant N36L MicroServer – this seems to be the best solution right now for a build-your-own NAS. It’s <$400, it's much faster than an Atom solution with about the same power consumption; supports 4 drives, sadly no hotswap. It’s actually about $50 cheaper at Newegg, but if you’re in CA, tax and shipping make it a wash w/ Amazon
  • 4 x WD 2TB Caviar Green WD20EARS drives – these drives are cool, quiet, and fast enough. I’ve been using them for years and haven’t had any problems. There was just a crazy $20/drive rebate that apparently I just missed, but jeez, $85 for 2TB? Just remind yourself how much that would have cost even 5 years ago.
  • 4GB 1333MHz DDR3 Non-ECC CL9 DIMM (Kit of 2) – HP thoughtfully includes 1GB of RAM with their MicroServer. Like the 160/250GB hard drive… it’s marginally useful. The server maxes out at 8GB, but 4GB for $44 is a pretty reasonable price. ZFS apparently really likes memory for cache, but lets get real. We’re not doing much mission critical work here.

I’ll post an update once everything’s up and running, including final kill-a-watt numbers.

Checkvist is my Task List

Like most geeks, over the years I’ve gone through a lot of a lot of task management tools (most successfully, OmniOutliner and TVO), only to go back to using text files (which unfortunately, tended to accumulate on multiple computers at multiple locations, I still have a few megs of files tucked all over the place). Lately I’ve been using a lot of Evernote, and as one of the few things that synced flawlessly (although Dropbox has really changed the game there), I used that for about a year or so as my main task “manager”, and used FogBugz for development-related stuff.

After the latest round being really unhappy w/ my setup, I took another look around what kind of online/shared options were available and stumbled onto Checkvist early last year. Although there are others that do more (Toodledo, and of course, Remember the Milk), Checkvist was the one that stuck the best, mostly because of its simplicity and speed. Also, probably the biggest factor (and a big contributing factor to the speed aspect) is that it can be almost entirely keyboard driven. It’s a moded editor with tons of chained shortcut keys. You can see the appeal to a longtime vim user. It also has decent multi-user sharing baked in from the start, which is nice for small teams (although there are a few things missing that have been stumbling blocks for adoption by my co-workers).

That being said, despite a few things that originally seemed like dealbreakers (lack of mobile app, for one), over the past few months, Checkvist has managed to take over as my primarily task manager. I attribute it primarily to the parts that it gets right: keyboard driven UI, search, list switching, expansion among them, that are just unmatched in anything else I’ve tried.

Here, BTW is my ongoing Checkvist Evaluation/Improvement List:

Checkvist’s closest peers are probably Todoist, which has a few nice touches, but seems to be inferior to Checkvist in just about every way, and Workflowy, which is conceptually very interesting (dynamic scoping/zooming, keyboard everything, and while slicker, is too limited to be really useful for me (even less metadata, no sharing). Also, I suspect that the lack of modalness actually makes the keyboard nav a lot more complex than it otherwise would need to be…

My Mac Apps

NOTE: I’m keeping a more up-to-date (and prettier version) of this list on Bagcheck.

One thing I’ve noticed having done a couple of reinstalls lately is that I’m using a lot less apps on my computer. I figure now (that I’m procrastinating) is as good a time as any other to write things down in case anyone finds it useful:

  • Google Chrome, Firefox 4 and Plainview
  • SIMBL and TerminalColours – pretty much the first thing I install, a must for seeting what you’re doing on a dark terminal background (I bump up green, blue, red, magenta and cyan). I’ve linked to the Network graph for TerminalColours, as the head seems to be ever-shifting
  • XCode
  • OpenTerminal – incredibly useful; I change the prefs to always open a new window and to open the selected folder
  • ClipMenu – the best clipboard history app I’ve found. I don’t use it’s snippets much, just increase the buffer to 50, increase the autosave frequency and show the last 10 inline
  • MacPorts – while others might have moved to Homebrew, MacPorts hasn’t failed me yet and I’ve invested the time to learning variants, upgrades, cleaning and activations. As long as it keeps working, I’ll keep using it. The first thing I’ll usually do is install python27 and python_select to python27. I finally learned my lesson not to use Apple’s built-ins. I also use MacPorts’ vim (+python27), curl (+ssl), and git-core.
  • Thunderbird – one day my Mail.app stopped working w/ GMail’s IMAP. I switching to browser-only, but that didn’t work well, especially w/ 5 GMail/GApps accounts. I’ve switched before, but after a year w/ Thunderbird 3, it looks like it’s taken for good. I have a few useful extensions, but by far the most useful is keyconfig, which I primarily to map the ‘y’ key to archive. I mostly use the widescreen view w/ Group By reverse chronological sorting in the unified inbox.
  • Alfred – I finally switched off of Quicksilver a few months ago. Now, there’s still some development going on, and it mostly works, but it felt like the right time to move to something a bit more vital. After trying Quick Search Box, I settled on Alfred mostly because I wanted a quick way to access individual preference panes. My launcher needs have always been pretty basic, I just wanted something that works and doesn’t crash much.

  • iStats Menu $ – For a long time I ran MenuMeters, which is great and does everything that I want. I mostly switched because iStats Menu has a nicer calendar than MagiCal, on par with the features I used from MenuCalendarClock (I can’t recommend the latter, as I registered, and shortly after a new version came out that required re-registering). Paying another $20 was a bit much for me.
  • Adium – the only feature I wish it had was cloud-based log syncing… hmm, I wonder if saving to Dropbox is a good option… I’ve use the Domo-kun notification icon for years, although the Totoro is pretty tempting. I make a lot of customizations to my Adium, but by far the most importan are deleting Growl notifications, turning off sounds, disabling the menu icon, and hiding messages and lists when backgrounded – basically, anything that would actually interrupt my work (I have my dock hidden by default, otherwise I would disable that stuff as well).
  • VLC – must have for MKVs and the like (I’ve used Perian in the past, just couldn’t be bothered to reinstall it)
  • Cog – I loathe iTunes, and Cog is the best lightweight music player I’ve found. The official release hasn’t been updated in a while. I use the latest nightly without problems. Vox is a slicker alternative, but doesn’t have playlist management or scrobbling (it does!). (Cog does via the Last.fm Scrobbler)
  • DTerm – I’ve yet to really start using it, but the copy results to clipboard seems really nice.
  • Alarm Clock – I found this a few years ago, and it does everything I want/need in an alarm clock.
  • Dropbox $ – I pay for the Pro account and use it everywhere. Here’s an interesting Quora question on why Dropbox is so much more popular than competitors. The simple answer is that it just works.
  • Carbon Copy Cloner – Its read-only image writing seems to work much better than SuperDuper‘s for some reason. With it, I can image a machine (11GB uncompressed) in about 5 minutes.
  • TeamViewer
  • 1password $ – this works I think as well as can be expected. LastPass is probably its only real competition. One slick thing is that if you’re syncing with Dropbox, the 1password’s license file will be automatically synced.
  • Evernote $ – It’s not perfect, but it does that one thing well that no one else did (sync!) With 2.0 (folders!), I find myself less angry at it now at least.
  • Gabble – a far better Yammer app than the AIR one
  • Sequel Pro – as much as I live in mysqlclient, Sequel Pro does some really convenient things (duplicating rows for example) and is a lot faster than MySQL Workbench.
  • TinkerTool – I only use this app to move the Dock (I run mine hidden on the top left side w/ only running apps), although I include it in the list because it’s does other neat stuff as well.

OK, not the shortest list, but compared to the number of apps I had installed before…

Looking Back, Looking Forward

Charlie Stross posted an interesting essay today, Reasons To Be Cheerful recapping some of the great things that have happened in the world over the past decade, primarily in the developing world. A great read, and honestly inspiring/heartwarming for the disheartened humanists. It’s easy to get overly cynical about it all. This is a good antidote.

That being said, I don’t think Charlie goes quite far enough. The essay starts framed by the thesis that in the world, things haven’t much improved, and the besides a few specific counterpoints about disease and the general march of technology, it feels like he gives up on really repudiating that thesis… for the developed world. And it’s easy to see why. In terms of general socio-economic trends, it’s hard to be all that positive. Things are downright unsettling heading towards dystopian. However, there’s at least one aspect, the very medium where we are commenting on that is worth, uh, commenting on.

Yes, the interwebbytubes, as Stross puts it, is quite a different place than it was at the beginning of the millennium. We are looking at a 2X adoption growth in developed nations (from plurality to supermajority, if not ubiquity). Worldwide, 2 billion people are now online. Beyond the quantitative changes, the qualitative changes are even more intriguing. In 2000 there was no Web 2.0. Blogging was in its infancy. Most of the things we take for granted online today were not invented yet. Among them: Wikipedia (2001), Facebook (2004), Google Maps (2005), Twitter (2006). I list these in particular because I don’t think there’s a day that goes by where I don’t use these particular services, but I’m sure that others have their own lists. Lest you think that this was a singular period of growth, I’ll throw in that the iPhone (2007) and iPad (2010) have kicked us into another era of hyper-growth that will be just as (if not more) life-changing.

We’re just starting to see what happens when the Internet starts engaging with us in a location/context aware fashion. We’re also starting to see what happens when Internet-style/scale dynamics are applied outside traditional consumer Internet contexts (e.g. Obama Campaign, 2008). In a historical scale, we’re still at the very beginning stages of figuring out what it means to live in a digital, massively inter-networked world, and similarly just starting to get a handle how that will change society (attention, communications and collaboration in particular).

All that’s a really long way of saying… well, there’s a pretty dang bright spot in the developed world too. One that has the potential of being turned into the shovel we need to dig ourselves out. So, here’s looking to the future. Happy New Year.

Winter Songs, Winter Tour

Just a few tracks, new and old, for the cold and dark.

Owen Pallett released the Export Demo EP for download on Soundcloud last week. (awesome!)

Oh, and thanks to FUELTV’s Green Label Experience for getting a whole bunch of songs stuck in my head.

Gawker Passwords, etc.

I have work deadlines, so I haven’t been able to been able to write a well constructed post about this, however, a few things:

  • To check if you had a Gawker account (there are 1.25M of them, so you might have one even if you didn’t realize it) I recommend: http://gawkercheck.com/. Note: even if your password wasn’t unhashed, consider it compromised. These passwords are encrypted with DES crypt, which is not adequate to stop attackers. The keyspace is too small. For more info on DES (and probably the best post-mortem so far), see this Forbes blog post.
  • This is as good a time as any to manage your passwords properly. A lot of people (including me) are using 1Password. It’s currently available as part of the MacUpdate December 2010 Software Bundle. LastPass also looks like a good solution and is free ($12/yr for mobile support). PwdHash and KeePass are also options.
  • According to the FAQ, Gawker claims to be sending emails eventually (and some people are doing so as well now). What I did last night, and maybe a good thing to do for your friends if you are an uber-geek is to go through your friends list and grep through the torrent database and them personally know if their account has been compromised, especially if the password has been unhashed.
  • Oh, lastly, if you’re a geek w/ your hash and want to check on whether it’s a reused password or not, you can pretty easily fire up a python shell and see if it matches:
    password = 'your_password'
    hash = 'your_hash'
    salt = hash[0:2]
    import crypt
    crypt.crypt(password, salt)

    If you’re not sure though, audit your passwords anyway when you have a spare hour or two. You’ll feel better, trust me.

Learning New Things

Today was an average afternoon – taking way too long to accomplish a seemingly trivial task, but looking up and learning a bunch of new things along the way. It seems there should be a better/easier (almost automatic, transparent) way to track the sources (links/pages), process (things tried) and results (code fragments)….

The basic goal in this case was to automate some execution of some javascript on a page. Because execution of the script caused a page load, it wasn’t a matter of writing the calls into the console. The faster and easier way would have been to write a Greasemonkey or Chrome Extension script (because there were timing issues, the script would have to write a time-based state file on actions), however, I figured I would try to see what kind of options were available with a control-script oriented model, as having that handy would be more generally useful in the future (more and more, straight mechanize is less useful as more JS proliferates).

Before getting started, I had to strip out just the lines that I wanted. I always forget how to do it, but that was a simple vim command lookup.

I looked at Selenium and Selenium RC, which probably would have worked fine (but that I didn’t use because I didn’t want to install extensions and the RC docs weren’t directly linked, but would have probably saved me time in the end).

Instead I decided to try out Watir (cross-platform, cross-browser, and my Ruby is rusty so this was a good excuse). I started out with SafariWatir, however, after a bit of poking, came up with a dead end on executing JavaScript. There’s a scripter object, but even after getting access to it via monkey-patching (did I mention my Ruby-fu sucks?), I was still getting errors and there wasn’t much help In general.

Instead of slogging through a potentially losing battle, I decided to jump ship to FireWatir. FireWatir uses JSSh, which communicates directly via JavaScript to Firefox, so it seemed like it might be a surer thing. My Firefox profiles were corrupted from my last system transfer, so there was a bit of messing with the profile folder until I gave up and started a new, but after that it seemed like I was home free.

Except, that when running js_eval, it turns out the scope that JSSh puts you in, isn’t in the document DOM, but rather the browser’s XUL DOM. For whatever reason, I couldn’t find a reference for even with the direct object type refereces (i.e. getWindows and getBrowser return ChromeWindow objects, which just don’t seem to have docs. Introspection via domDump() or inspect() just returned a huge amount of stuff to go through). Luckily, while searching, one of the results that turned up was a StackOverflow question on firewatir + jQuery which answered the question – ChromeWindow.content gets you into the HTMLDocument DOM. I’m a bit mystified why this isn’t in the firewatir or JSSh docs, as this seems to be one of the most common things that people would want to do, but well, that is the life of the developer…

Fall Music

One of the little pleasant surprises as I settle into a routine in LA has been rediscovering how great KCRW is. My drive usually has me catching some of Morning Becomes Eclectic on the way in, and Chocolate City or Raul Campos in the evening. Garth in particular has a knack for playing some killer cuts (that Shazam invariably does nothing for and where I have to manually remember the time and look up later).

It’s been a long time since I’ve posted anything. So, in the spirit of holiday procrastination, here’s a little mini-mix of stuff I’ve been enjoying recently: