My Amnesia Fortnight 2014 Picks

Posted on by lhl

Today’s the last day to vote, so figured I’d post my picks for the 2014 Amnesia Fortnight.

It’s not the most ambitious idea, but I think this is the most likely of Pendleton Ward’s pitches that will be a fun game in 2 weeks:

There are tons of great pitches (wish I could vote for more than 3!) but here were the ones I ended up picking, weighted against cool Rift experiences.

2014 Personal Data Storage

Posted on by lhl

A few years ago, I went ahead and built a NAS w/ 4x2TB drives, about 7TiB of usable storage (interestingly enough, 2TB drives are the same price as they were 3 years ago). I had a separate 2x3TB external setup for backup, as well as way too many external drives (about 10TB of loosies). Later, I also inherited another microserver (w/ the same 4x2TB setup) from work as we switched to two Synology DS1812+‘s in the office. And I also have an old Drobo Gen 2 that’s still ticking along.

As you might surmise, my personal storage situation has been… loosely managed. I have a spreadsheet that helps a bit, but it’s always a mess as I run out of space on a drive and the nice clean schema gets all out of whack. Recently I decided to try to get a better handle on things, and acquired 5x4TB of external drives to help rejigger things. This was a good start, but as the drives on my main NAS were giving up the ghost, and as I was digging through my backups, backups of backups, and other duped files, I realized I probably needed a new NAS to help work things out.

The state of dedupe/file organizing tools is the topic of another long blog post. I’ve used Disktracker off and on, which is ancient, and not quite what I want, but seems to indexing tens of millions of files the best. The general state of desktop file management/indexing/duping is quite sad however, especially if you’re interested in consolidating and comparing entire folder structures/partial drive backups. (as far as I can tell, there’s no good tool for that).

In any case, after a late night of clicking around, I decided that as the Drobo 5N was pretty reasonably priced, and that as my old Drobo was still working fine after almost 6 years, and people online (and Drobo themselves) seemed to think that the performance was much improved, that it might not be a bad way to go. I picked one up about two weeks ago.

So, how is it? Well, for general large file copying, it seems fine (I can get about 80MB/s transfers on my netwrok). However, my primary use case involves lots of rsyncing and dealing w/ small files, and here, the Drobo 5N is, as far as I can tell, pretty pathetic. You can read some more details, but even with the mSATA accelerator (I am using a 128GB Plextor M5M which is specc’d for 76K write IOPS), rsyncing maxes out at all of about 700 IOPS.

I *just* finished rsyncing my Drobo Gen 2 yesterday, and I have a 4TB drive that will hopefully be done in the next day or so.

Still, this setup I think is good enough as I prep for moving about. We’ll revisit this in a few years, I guess.

2013 Review in Tech

Posted on by lhl

I’ve been a bit under the weather the past couple days (the dangers of hanging out near other peoples’ little germ factories (aka kids)), but I wanted to post some of my thoughts about the year in tech. The last time I did that was probably a few years ago (related).

Over the weekend a friend was going on about how this year was a crappy one for “tech” (echoing the sentiments of those crappy articles floating around, but more along the lines of lack of ambition/innovation) which I strongly disagreed with. I think the kernel of truth there is that the SF/SV tech scene is definitely caught up in a weird spiral of chasing/making less and less interesting mobile/social apps, so of course from that vantage point, it’s going to seem terrible, but from the outside, things are… pretty interesting.

  • NSA Leaks – In some articles, this was cited as some negative development, but Edward Snowden’s leaks revealed (and continues to reveal) how much the world has been changed by technology and hints at some of the implications that both as technologists and end-users, we’ve been oblivious to. It’s certainly the biggest tech story of the year, and has profound/deeply unsettling implications. It’s also kicked off a number of new projects, and made a lot of techies think harder about the things they’re working on. I think that in coming years, the world and the tech industry in particular will be better for it.
  • Bitcoin – Bitcoins, alt-coins, cryptocurrency. While it’s been building up steam, this is the year that it boiled over and it’s another development that’s more than a little world-changing. I mentioned it briefly in a rebuttal comment I made on Charlie Stross’s blog post Why I want Bitcoin to die in a fire (I don’t know about his conclusions, but on just about every point of fact supporting his reasoning, he cited inaccurate/just plain wrong sources), but regardless of what the eventual value of BTC ends up as (which this year was driven mostly by the Chinese, not media hype), whether it’s $10K/BTC or $0/BTC, Bitcoin has not only served as a solid proof of existence for the viability of truly P2P digital currency, but has also laid down a protocol/framework that makes it trivial to create your own. The classes of problems that can be solved by a distributed public ledger are numerous… There’s some more thinking I need to put on that.
  • Tesla – Finally, something out of the Valley. Between Tesla, SpaceX, and SolarCity, this was a huge year for Elon Musk, and they seem to all be converging into some techno-utopian vision that’s quite honestly, a rather refreshing respite from the totalitarian surveillance state, increasing economic disparity and general grim meathookiness going on elsewhere. If you aren’t excited about what Tesla has been up to this year, maybe you just aren’t that into tech.
  • Robots – Google’s buying spree was pretty well reported, but less well covered was Schaft’s (one of the Google acquisitions) performance at the DARPA Challenge. It scored 27 out of a possible 32 points on the challenges. That’s 84%. This year has shown some tremendous accomplishments in robotics on just about every level, most interestingly/disturbingly in drone-tech. If you haven’t read this recent brief, but intense editorial in the Guardian this past week, btw, please do: I worked on the US drone program. The public should know what really goes on
  • Kickstarter – I’ve been active (maybe a bit too active, seeing as I started getting KS spam this year) on Kickstarter this year. It’s not new, but it’s certainly gained even more steam in 2013, and I don’t see it decelerating. Kickstarter seems to be increasingly, one of the more important tools helping the Maker/DIY movement grow.
  • 3D Printing – speaking of which, another not quite new, but thought I’d mention it, we finally got our Replicator 2 in the office this year, and it’s been incredibly useful. It’s also very close to being consumer friendly/ready. Like, say if the platform would self calibrate and if the prints were a little easier to peel off… But still, getting a 3D printer is now cheaper than an office laser printer was a decade ago.
  • Quantified Self – In some ways, still nascent, but I got my Basis Watch, and I’ve been trying more than a few autologgers/aggregators released this year (Saga, Memoir, Heyday among others), but I think we’re seeing some really interesting first steps into pulling together both the data exhaust we’re already generating and combining that usefully with other things we’d like to track (beyond fitness trackers, things like the Automatic car tracker). One thing’s for sure though, things are just getting started
  • VR – I admit, my Oculus Rift Devkit sat unloved and completely unopened for months. It’s been a hectic year. I did finally get around to break it out and try out about a dozen demos, and it was great, and also left me motion sick the rest of the night (I’ve played FPS/TPS’s for decades w/o problems). Carmack’s full time commitment and the news coming out of the community has me hopeful that they’ll have that problem licked, but it’s been pretty exciting following along. For those interested in what’s going on, I recommend Road To VR
  • Open Hardware – this is an ongoing thing that isn’t new in 2013, and has also been greatly helped by Kickstarter, but there’s just a ton of interesting stuff happening in the cheap microcontroller world akin to the early web days. I’ve been poking around with a lot of this stuff, but this year, got pretty serious about it, doing a fair amount of soldering, exploring/evaluating pretty much every single ARM dev board around, and getting my first PCBs printed. Again, we’re going to see a continued proliferation of interesting hacks/automation/sensors as it gets increasingly cheaper and easier to program the world

I think most of these things point to how wrongheaded talking about these things in context of a year are though – tech is incremental, and it’s hard work. You can bet anything that’s being announced, let alone making a big splash probably took quite a bit longer than a year to get there.

I’ll also link to Some Notes on Labor, Technology and Economics that I wrote about earlier this year. 2013 started out on sour note and what’s been going on in the world this year has definitely given me some pause.

We’re none of us getting any younger, and the pace of innovation continues to increase though, so here’s to the next year. Let’s hope we can make it a better world.

ADDENDUM: Putting this here since it’s related. Not really an innovation per se, but a tech problem that may be reaching a breaking point – we continued to see bigger and bigger data breaches (Adobe, Target, SnapChat) by cyber-criminals. Will 2FA finally replace Passwords? Is there a different security model that can more effectively handle APTs/inevitably compromised networks? Is there a way to expire/invalidate leaked data or will fraud models improve enough that it’s OK that black-hats and script kiddies around the world trade your personal info? In traditional security, the deck is always stacked against the defense, but it makes me wonder if there’s not a way of changing that – after all, the physics of software (if not software engineering) are malleable…

2013 Geek Reading:

Pythonista and iOS Automation

Posted on by lhl

While in general, iOS is pretty nifty, it has some pretty annoying limitations, particularly in regards to inter-app communications (more specifically, the lack thereof). While Pythonista doesn’t complete fix this terrible situation, it does provide some really interesting workarounds.

I’d heard of Pythonista when it first launched, but mostly ignored it since it just seemed like just another Python REPL. Now, it is a Python REPL (2.7) and editor, and it also includes its own 2D graphics and multitouch libs so you can make simple interactive apps with it. Where it starts to get really interesting is that it also includes a bunch of modules, that exposes the iOS clipboard, contacts, location, notifications, etc.

Mostly importantly, it has urllib and webbrowser module support that supports iOS URL callbacks. Pythonista itself supports its own URL scheme of course, lending itself to being called remotely.

Another useful app to use in conjunction with Pythonista is Agile Tortoise’s Drafts. It’s a text editor built specifically to interface with other apps and can serve as an easy briding tool.

There are a fair number of tutorials/guides/scripts available online.

First, some general reference:

And some useful scripts:

Specific HOWTOs:

Targus AKF001US Foldable Bluetooth Keyboard Review

Posted on by lhl

I just got a Targus Universal Foldable Keyboard for Android Devices, Black (AKF001US) from Amazon (about $50), and thought I’d write a quick review.

First of all, a picture of the keyboard layout. I wasn’t able to find a single image showing the actual key layout, even on Targus’s site (WTF), so here it is:

Targus AKF100US Foldable Keyboard

I looked at a bunch of other folding Bluetooth keyboards, notably the Perixx PERIBOARD-805L and pretty much every terrible OEM/off-label option I could find, but dismissed most immediately due to bad layouts. Actually, there’s one folding keyboard that has a great layout except it’s, um, a full 104-key with number pad.

(The Periboard 805L looked promising, but the combination of shifted numbers (due to a mysteriously oversized ESC key), tiny Backspace, and more seriously, the arrow key in between the a tiny right shift key made me reconsider. There’s a review/unboxing vid online as well as plenty of useful reviews on Amazon.)

The Targus’ layout isn’t bad except for the ‘B’ key which is on the wrong side of the 6mm folding gap. This is mega annoying, although my fingers were able to physically jump the gap without too much trouble, when touch typing I pretty much have to type the key twice – once missing it, and a second time reaching for it. It’s a major bummer.

The keys themselves are smaller than full-sized. They are about 15mm, or ~80% regular size (there is a 2mm gap between keys). With a folding keyboard, I would have much rather traded overall size to get 90% or 95% keys. It’s a bit cramped but I’m able to touch type on this, however people with bigger hands probably won’t be able to. The keyboard/key feel is as expected, rather soft/spongy with limited travel, but it’s better than typing on an 8″ touchscreen.

There’s a button to press to unfold, but there’s no mechanism for locking it in a folded position. You won’t be able to use it without a flat surface for it. In fact, there’s a slight “springiness” so that it won’t stay open when held vertically, although this isn’t a problem when flat. There’s a power switch, which is fine, a light rechargeable battery, and a mini USB port for charging.

This is a Bluetooth 3.0 keyboard and I and despite being an “Android” keyboard, I had no problems pairing it with my iPad rMini. It’s responsive, and seems to work well. Typing on the keyboard will wake/unlock the device. Note, iOS 7 (I’m on 7.0.4) currently has some majorly bad and intense Bluetooth Keyboard issues (see discussion). Basically, some combination of turning on/off the keyboard will cause iOS to hard lock, requiring a Power+Home Button reboot. I personally experienced this once already.

Here’s how it looks with the iPad rMini (the keyboard comes with an external case that doubles as a stand, although it’s quite bulky, so if you already have a smart cover like I do, you’d probably just toss the case):

Targus AKF001US Foldable Keyboard

Besides the terrible locking up issue, it seems pretty well behaved. Just a few notes:

  • Most of the special keys work: Home functions as the home key, Search brings up spotlight, ‘Select Left’ and ‘Select Right’ navigate the cursor, the Menu key also works as a home key (I can triple-tap with the Menu key but not the Home key), and the volume/media keys seem to work as well
  • While most of the regular typing keys work (including CTRL-A, CTRL-E, Page Up/Dwn), iOS keyboard support isn’t fantastic. Most notably, you can’t use keyboard clipboard shortcuts, and it’s often impossible to switch focus, scrolling, or using select boxes without stabbing at the screen. Also, while it has an ALT key which can be used to easily type all the regular extended key shortcuts, I couldn’t find a CMD key equivalent, so you’ll miss out on some shortcuts.
  • There is a real-deal ESC key on the keyboard. Unfortunately I wasn’t able to map it w/ Applidium’s Vim, although testing it in iSSH, it did work on a terminal. Local editing of course sucks without easy access to the filesystem.

    • Right now, I’m not exactly sure if I’ll be keeping this keyboard. I think it may be the best folding keyboard available right now, and if I keep it it’ll be to try to take it apart and improve it (I’m almost sure that I can modify the case to remove the folding gap. And I think I can solve that and the locking issue with a tyvek band or something), but otherwise, man, that B key. That, and the just too small size are the real issues for me. Sometimes you have to wonder what the hell these people are thinking. Do they even use their products?

    One thing I haven’t seen is any Bluetooth Low Energy keyboards (they should be coming soon) which may or may not be relevant. Also, I would love to see more keyboards like Logitech’s K810/K811 that support pairing and 1-click switching between multiple devices.

    Oh, and here’s a size comparison of the keyboard folded up stacked with my iPad rMini and iPhone 5s for a size reference:

    Targus AKF001US Foldable Keyboard Size Comparison

    Targus AKF001US Foldable Keyboard Size Comparison

    UPDATE: I’ve found a portable foldable keyboard that is close to perfect:

    • Almost completely standard (US ANSI) keyboard layout (except for the up arrow key is between the right shift). Importantly, all the middle keys are regular sized and there is no space between the halves when folded out (via a very slick sliding mechanism); also, standard size 18mm wide keys
    • BT 3.0. Bluetooth HID w/ pinless pairing. Can be paired with up to 4 BT devices and searches in order of last found
    • Lockable when opened (also has an optional slideout stand)
    • Turns on when opened, turns off when closed
    • Uses two AAA batteries inserted on the top-right

    So what’s the catch? As far as I can tell, the keyboard, the Reudo RBK-3200BTi is only available in Japan. (I bought it from Amazon.co.jp for ￥6,581 (~$65). It’s made in China, but doing a search, I haven’t been able to find anyone selling the same design.

More on OS X Mavericks Power Usage

Posted on by lhl

The “Energy Impact” tab in OS X 10.9 Mavericks’ Activity Monitor is exactly what I would’ve wanted when I was testing my new Macbook Air a few months ago. Although, one thing I’ve found is that the “Avg Energy Impact” (it averages over the past 8 hours) isn’t actually a very useful/stable number, at least for measuring something that can have a very variable/spiky usage, like browsing with lots of tabs.

I knew if it was in Activity Monitor though, that the raw data must be available, and after some fruitless online searching, I just dug around my system and ended up finding some cool stuff in /usr/bin. First is a script called power_usage.sh that dumps out a report of lots of technical details.

Here’s a small sample of the output:

**** Battery and backlight usage ****

Backlight level: 862 (range 0-1024)


**** Network activity ****

out: 1.00 packets/s, 94.29 bytes/s
in:  1.50 packets/s, 212.53 bytes/s


**** Disk activity ****

read: 0.00 ops/s 0.00 KBytes/s
write: 5.49 ops/s 32.70 KBytes/s

****  Interrupt distribution ****

CPU 0:
	Vector 0x46(SMC): 2.99 interrupts/sec
	Vector 0x49(MacBookAir6,1): 0.50 interrupts/sec
	Vector 0x92(IGPU): 107.76 interrupts/sec
	Vector 0x98(ARPT): 9.48 interrupts/sec
	Vector 0x9e(SSD0): 3.99 interrupts/sec
	Vector 0xdd(TMR): 691.96 interrupts/sec
	Vector 0xde(IPI): 8.98 interrupts/sec
CPU 1:
	Vector 0xdd(TMR): 6.49 interrupts/sec
	Vector 0xde(IPI): 3.99 interrupts/sec
CPU 2:
	Vector 0xdd(TMR): 16.46 interrupts/sec
	Vector 0xde(IPI): 8.98 interrupts/sec
CPU 3:
	Vector 0xdd(TMR): 8.48 interrupts/sec
	Vector 0xde(IPI): 2.49 interrupts/sec



**** Processor usage ****

Intel energy model derived package power (CPUs+GT+SA): 0.58W

LLC flushed residency: 93.2%

System Average frequency as fraction of nominal: 85.75% (1457.75 Mhz)
Package 0 C-state residency: 93.81% (C2: 13.45% C3: 0.07% C6: 0.00% C7: 80.29% C8: 0.00% C9: 0.00% C10: 0.00% )

Core 0 C-state residency: 96.16% (C3: 0.00% C6: 0.00% C7: 96.16% )

CPU 0 duty cycles/s: active/idle [< 16 us: 20.45/11.47] [< 32 us: 7.98/15.96] [< 64 us: 70.34/13.47] [< 128 us: 23.45/11.97] [< 256 us: 14.97/6.49] [< 512 us: 18.46/2.00] [< 1024 us: 4.99/4.49] [< 2048 us: 3.49/8.98] [< 4096 us: 2.00/12.97] [< 8192 us: 0.00/24.45] [< 16384 us: 0.00/26.94] [< 32768 us: 0.00/26.94] 
CPU Average frequency as fraction of nominal: 77.55% (1318.39 Mhz)

CPU 1 duty cycles/s: active/idle [< 16 us: 15.47/4.49] [< 32 us: 1.00/2.49] [< 64 us: 2.99/2.00] [< 128 us: 1.50/0.50] [< 256 us: 4.49/1.00] [< 512 us: 1.00/1.50] [< 1024 us: 1.00/1.50] [< 2048 us: 0.50/0.50] [< 4096 us: 0.00/0.50] [< 8192 us: 0.00/2.49] [< 16384 us: 0.00/2.99] [< 32768 us: 0.00/0.00] 
CPU Average frequency as fraction of nominal: 83.51% (1419.63 Mhz)

Core 1 C-state residency: 96.81% (C3: 0.01% C6: 0.00% C7: 96.81% )

CPU 2 duty cycles/s: active/idle [< 16 us: 68.35/4.49] [< 32 us: 8.98/5.49] [< 64 us: 5.49/17.96] [< 128 us: 6.49/12.97] [< 256 us: 7.98/13.47] [< 512 us: 2.99/7.48] [< 1024 us: 2.00/2.49] [< 2048 us: 1.00/2.49] [< 4096 us: 0.00/0.50] [< 8192 us: 0.50/7.98] [< 16384 us: 0.00/9.98] [< 32768 us: 0.50/7.98] 
CPU Average frequency as fraction of nominal: 94.05% (1598.86 Mhz)

CPU 3 duty cycles/s: active/idle [< 16 us: 9.48/1.50] [< 32 us: 2.00/2.49] [< 64 us: 0.50/2.00] [< 128 us: 3.99/0.00] [< 256 us: 1.00/1.50] [< 512 us: 1.50/0.50] [< 1024 us: 1.00/0.50] [< 2048 us: 0.00/0.00] [< 4096 us: 0.50/0.50] [< 8192 us: 0.50/0.00] [< 16384 us: 0.00/2.99] [< 32768 us: 0.00/1.50] 
CPU Average frequency as fraction of nominal: 96.73% (1644.44 Mhz)

**** GPU usage ****

GPU 0 name IntelIG
GPU 0 C-state residency: 99.73% (0.00%, 99.73%)
GPU 0 P-state residency: 1100MHz: 0.00%, 1050MHz: 0.00%, 1000MHz: 0.00%, 950MHz: 0.00%, 900MHz: 0.00%, 850MHz: 0.00%, 800MHz: 0.00%, 750MHz: 0.00%, 700MHz: 0.00%, 650MHz: 0.00%, 600MHz: 0.00%, 550MHz: 0.00%, 500MHz: 0.00%, 450MHz: 0.00%, 400MHz: 0.00%, 350MHz: 0.28%, 300MHz: 0.00%, 250MHz: 0.00%, 200MHz: 0.00%
GPU 0 average frequency as fraction of nominal (200.00Mhz): 0.49% (0.98Mhz)
GPU 0 GPU Busy 0.28%
GPU 0 FB Test Case 0.00%

Most of the best stuff is coming from /usr/bin/powermetrics. Here's the man page for powermetrics. There's a lot of options to go through.

I couldn't get some of the --hide flags to work, but here's a quick command to pull just the power score from some apps. Suffice to say, the possibilities for this in doing power comparison are quite tantalizing:

powermetrics -i 1000 --poweravg 1 | grep 'Average cumulatively decayed power score' -A 20

Example output:

--
**** Average cumulatively decayed power score ****

                          	15 sec    	1 min     	5 min     	15 min    	1 hr
[26367]systemstats         	   157.144	   43.3773	   15.5405	
[0   ]kernel_task         	   74.2694	   74.8748	   47.8391	
[-1  ]DEAD_TASKS          	   48.8577	   24.5274	   6.72759	
[123 ]WindowServer        	   30.0174	   35.1636	   21.6378	
[388 ]iTerm               	   21.9219	   20.0339	   12.7875	
[26917]Airmail             	   18.7339	   19.6338	   9.45401	
[41829]Google Chrome He    	   10.6604	   7.92617	   4.61903	
[42323]com.apple.WebKit    	   10.1828	   17.2152	    13.345	
[41206]Safari              	    9.1336	   16.8081	   10.7422	
[42352]Activity Monitor    	   7.60748	    6.4298	   3.80237	
[96  ]hidd                	      7.59	   7.70318	     4.438	
[42612]powermetrics        	   4.74147	   4.81903	   3.23933	
[37370]SystemUIServer      	   4.36079	   4.44251	   3.02667	
[159 ]sysmond             	   3.80035	   2.93568	   1.85484	
[442 ]Moom                	   3.53523	   2.59021	   1.21046	
[389 ]Dock                	   3.43152	   2.84213	   1.58266

I also made another cool discovery for power measurement. Originally I first tried to see if Apple's top had the power information, but sadly, it appeared like it didn't. Well, it turns out that (and the top man page) was wrong - my 11" MBA's screen simply wasn't big enough (even maximized) to show the POWER column, opening top open on my bigger desktop screen showed that this wasn't the case!

So here's an equivalent "power" command for top:

top -stats pid,command,power -o power -l 0

And the output looks something like:

PID    COMMAND          POWER
123    WindowServer     31.1
388    iTerm            25.7
42689  top              10.5
96     hidd             2.9
26917  Airmail          1.4
37370  SystemUIServer   0.4
45     fseventsd        0.3
85     mds              0.1
42323  com.apple.WebKit 0.1
31927- Dropbox          0.1
31997  WebKitPluginHost 0.1
453-   Google Drive     0.1

It'd be pretty cool to make an open source power measuring/graphing framework combined w/ a browser automation test suite. I might get to it one day, but maybe someone w/ some more free time might feel like doing it sooner? Just putting it out there.