It’s been a long while since I’ve spent much time thinking about file compression tools, but I was running out of space on one of my smaller SSD’s and the culprit was mainly lots of large multi-GB TIFs and PSDs that had accrued in our Lensley Dropbox.

My first thought was xz, since it compresses super well, but I was growing old waiting for files to compress and decided to switch. lrzip seemed like a good option, and performance was much better than xz, and threaded (and with multi-file support and a nice progress UI to boot), but I also happened on another compressor shoot-out page (if you’re interested in this topic, definitely read this, there’s a lot more testing and nice pros/cons list for the apps) which concluded that lbzip2 was a good choice.

I gave it a spin, and sure enough lbzip2 is great. With defaults, it loses a few percentage points when compressing vs lrzip, but it also compresses on average twice as fast, with the kicker being that its archives are fully cross-compatible with bzip2, so pretty much any system can access the archives by default. I’ll be aliasing bzip2 to lbzip2.

(If speed isn’t as important (batch compression) and for max compression of large files, lrzip is probably better. Although a lot of people have been using xz, even with multithreaded support, I still didn’t feel performance was great so for personal use, I’ll probably end up mostly using lbzip2 or lrzip from now on.)

We have relatively modest AWS bills w/ Lensley, but it’s been inching upwards and broke $400/mo at the beginning of the year. Originally, I thought this was mostly just due to ever increasing S3 usage, but it turns out that wasn’t the case. After going through and trimming things, we’ll end up saving about $250/mo, which isn’t a huge absolute amount ($3K/yr) but was over 60% of our monthly.

• Obviously there were a couple EC2 instances hanging around that we didn’t need. One was a bit pernicious since it kept regenerating (not a Spot Request, but actually due to an old Elastic Beanstalk test. Oops.)
• Our Reserved Instances had lapsed, so everything was 50% more than it should have been. Double oops.
• Our instances were running old and busted m1’s – with our usage I was able to switch m1.smalls to t2.micros and m1.medium to t2.smalls, cutting costs by 2/3’s (w/ equal Reserved discounts).
• Note: T2s are HVM only. Migrating from PV to HVM wasn’t too bad, although I also took the opportunity to switch one machine to the new 16.04 LTS (if you don’t have to, don’t – besides some package migrations, there’s also a killer kswapd bug that pegs the CPU)
• It’s worth noting here that one of the instances was running EBS Magnetic storage. Obviously worth snapshotting and switching over to gp2.
• We’ve been paying for an ELB and MultiAZ instances but honestly, have never had to use the failover. At the risk of jinxing myself having typed that, I just got rid of them.
• A few years ago I switched to a Multi-AZ RDS instance for our database. At the time this seemed like a good idea, and while I never had to worry about things, w/ the rollout of gp2 (SSD) EBS w/ plenty of IOPs, it seemed maybe a bit wasteful for minimal convenience. Switching from Multi-AZ db.m1.medium to a t2.medium w/ a 100GB gp2 EBS volume lowers our monthly cost from ~$160/mo to $33/mo
• Clearing out unused EBS volumes/snapshots, ofc

A big chunk of our savings were the result of getting rid of some redundancy, but since I can count the number of single-AZ outages (we’re in us-west-2 in Oregon) we’ve been affected by over the past 3 or 4 years on a single hand and there’s nothing super-mission critical that can’t live with a few minutes of downtime.

I run a lot of browsers – I’ll usually have 3-4 running, a mix of Chrome, Canary, Safari, and Firefox Nightly. With Mavericks, I switched to Safari as my default browser on my MBA due to its power efficiency.  Unfortunately, Yosemite breaks the SIMBL plugin I was dependent on, so it was time to move on.

Chrome has been getting sluggish and I’ve really been liking what Firefox has been up to, but the latest Nightly builds have been not so dependable (I blame e10s but maybe that’s unfair) and since I’m traveling again, daily 90MB downloads isn’t ideal, so I decided to give Firefox Developer Edition a shot.

Turns out, it’s pretty great! It has a dark simple, theme, by default. Is pretty snappy, and the developer tools look great (although at this point I’m so used to Chrome’s keybindings that it’s been a bit awkward switching).

The one fly in the ointment was that 1Password wasn’t playing nice. Luckily, there is a solution. Just upgrade to the latest beta extension and the latest beta version of the app and it’ll work.

If you use Evernote, you’ll also want the beta Clipper that brings it to parity w/ Chrome and Safari.

Lastly, one of the things that I really got spoiled by was Chrome’s particularly elegant “hold CMD-Q to quit” option. While, ever so slightly less elegant, meta-q-override/warn-before-quit does the trick.

I’m currently using Firefox Developer Edition as my new default browser.

Ello blew up this week. It’s new and shiny and does some interesting things. That being said, it’s not where social networking or how we use the Internet needs to go.

If you want more reading:

• Andy on VC funding
• Aral Balkan on Sustainability/Economic logic of VC funding
• Quinn on Social Networks
• Anselm on decentralization and protocols

I had posted some of my own initial thoughts, which is that the ideal social network should be end-user controlled and distributed and decentralized. A natural pre-condition is there should be an open protocol, but it’d be worth fleshing out the type of functionality that’s required (I’ll have to revisit some relevant thinking I did in the early 2000s in decentralized SNSs, the mid-2000s on permeability/privacy, and the late-2000s on Y!OS).

FWIW, the more interesting social networking-related project I discovered is an open source, decentralized, massively-distributed 3D simulation engine called Lucidscape. It is explicitly designed for an open metaverse. (See also: Open Cobalt née Croquet)

In Python, you can set most request timeouts w/ socket.setdefaulttimeout(). In recent versions, urllib2 has also added a timeout field to urllib2.urlopen(). So far so good, right?

Unfortunately, while these work fine when looking up IPs or domains in /etc/hosts, this fails miserable when querying a FQDN as you’re at the mercy of socket.gethostbyname() and your DNS resolver which does not let you adjust the timeout. On my Mac this defaults to 30 seconds. It’ll ruin your day, really. (A good recent thread, old summary)

This is a somewhat common problem and you can see a lot of various workarounds (using signals didn’t work for me). The proper modern way is to probably using multiprocessing with a join(timeout) (sample) but that seemed awfully wordy, so here’s my simple one-line hack that I ended up with instead:

subprocess.check_call(['/sbin/ping','-t','1','FQDN'])

Just set 1 to the timeout you want. It’s hacky, but it works and it’s much easier and shorter – a one liner in a try block without any other libraries. Another advantage this has is that it works as it should both with DNS and mDNS (zeroconf) without any additional lookups. I’m using this for finding local machines so this is quite useful.

Some extra references:

• pybonjour, zeroconf – a couple zeroconf libs that are way overkill for what I wanted to do
• monkeypatch/custom dns – creative way to solve this
• another monkeypatch example
• dnspython – for custom resolver
• timeout w/ subprocess – you’ll need the subprocess32 backport if you need this
• tornado-dns – async dns w/ Tornado (fyi Tornado blocks on DNS for async_http calls by default – still as of late 2012)

UPDATE:

So, at the end of 2016 I encountered this problem, and decided that I could do better, especially because I wanted to do a sub-second query. I decided that of course the way to go would be to use concurrent.futures, but that was actually wrong, it turns out. When you call whether on the ThreadPoolExecutor or ProcessPoolExecutor version, it still waits for socket.gethostbyname() to finish. Here’s the simplest code that I implemented that worked:

from   multiprocessing import Process, Queue
def dns_lookup(host, q):
  try:
    socket.gethostbyname(host)
    q.put('OK')
  except:
    pass

q = Queue()
p = Process(target=dns_lookup, args=('example.com', q,))
p.start()
time.sleep(0.4)

if q.empty():
  p.terminate()
  print('dns timeout')

Using the multiprocessing library turns out to be the way to go because the terminate() function actually works like you expect it to, killing with extreme prejudice and w/o too much extra code. Hope that helps anyone dealing with the same problem.

After 511 days of uptime, I decided it was time to bite the bullet and do a version upgrade. The `do-release-upgrade` command did what it said on the tin, and the upgrade from 9.10 to 10.04LTS was pretty straightforward (some downtime waiting for the disk to fsck, and requiring ops to manually reset). Unfortunately, the upgrade made WordPress pretty unhappy. Some combination of WP, APC, and potentially WP Super Cache? Instead of using Ubuntu’s APC (3.1.3p1-2) I switched to a pecl install (3.1.9). This didn’t solve things, so I bumped up the apc.shm_size to 128M…

I’ve been lackadaisical lately w/ my sysadmining, but with the unfriendly waters, I took some time to tighten the ship up a bit. I probably be publishing a little “hardening Ubuntu for really lazy/busy devs” guide soon.