Classic Code

I’ve been syncing files/copying drives onto my new NAS for the past few days and I may actually end up running a bit low on space (currently up to 26/40TiB, 13M+ files and counting).

While my main plan is to do mass deduplication as a part of a larger effort (there are multiple backups/copies of many of the files I’m dumping onto the NAS) if I run out of space I may have to do some manual lookups, which will probably involve using something like fdupes.

One interesting thing looking at the fdupes repo is that it’s about to turn 19 years old soon (actually pretty close in age to this blog), maintained basically by a single author over the years. It’s at version 1.6.1 currently.

Anyway, just thought that was a neat little thing. When you think about it, all computing is about people dedicating time and energy towards building this larger infrastructure that makes the modern world possible (one bug at a time), but there are tons of these small utilities/projects that are stewarded/maintained by individuals over the course of decades.

On a somewhat related tangent, a small anecdote which I don’t think I ever posted about here, but a few (wait, 8!) years ago WordPress asked me if I’d re-license some really old code (2001) I wrote from GPLv2 to GPLv2+. It turns out this code I wrote mainly as a way to learn Cold Fusion (which also still runs in some form in Metafilter I believe) lives on in the WP formatting code and gets run every time content is saved. It’s some of my oldest still-running code (and almost certainly the most executed), and what’s especially interesting is that it pretty much happened without any effort from my part. I put it online on an old Drupal site that had my little code projects back in the day and one day Michel from b2 (proto-WP) dropped a line that he had used the code. The kicker to the tale is that I switched majors (CECS to FA) before taking a compilers class or knowing anything about lexing/parsing (which, tbf, I still don’t), so it’s really just me writing something from first principles w/ barely any idea of what I was doing. And yet, if the stats are correct, probably a third of the world’s published content has been touched by it. Pretty wacky stuff, but probably not such an uncommon tale when you think about it. (I’ve also written plenty of code on purpose for millions+ of people to use; it’s one of the big appeals of programming IMO.)

Tangent two: eventually I will publish my research into file and document crawlers, indexers, management systems. I’m trying out Fess right now, which was pleasantly easy get running, and plan on trying out Ambar and Diskover. I have an idea of the features I actually want and they don’t exist, so it’s likely that I’ll end up trying to adapt/write a crawler (storage-crawler? datacat? fscrawler?) and then adding what I need on top of that.

2018 DIY ZFS NAS (Disaster)

My last NAS build was in 2016 and worked out pretty well – everything plugged in together and worked without a problem (Ubuntu LTS + ZFS) and as far as I know it’s still humming along (handed over for Lensley usage).

Since I settled into Seattle, I’ve been limping along with the old Synology DS1812+ hand-me-downs. They’re still chugging along (16.3TB of RAID6-ish (SHR-2) storage), but definitely getting long in the tooth. It’s always performed terribly at rsync due to a very weak CPU, and I’ve been unhappy with the lack of snapshotting and inconvenient shell access, among other things so, at the beginning of the year I decided to finally get moving towards build a new NAS.

I wanted to build a custom ZFS system to get nice things like snapshots and compression, and 8 x 8TB w/ RAID-Z2 seemed like a good perf/security balance (in theory about 43TiB of usable space). When I started at the beginning of the year, the 8TB drives were the best price/perf, but w/ 12TB and 14TB drives out now, the 10TB drives seem to have now taken that slot as of the time of this writeup. I paid a bit extra for HGST drives as they tend to historically have been the most reliable according to Backblaze’s data, although it seems like the newest Seagate drives are performing pretty well.

Because I wanted ECC and decent performance (mainly to run additional server tasks) without paying an arm and a leg, and I had a good experience with first-gen Ryzen on Linux (my main workstation is a 1700 on Arch), I decided that I’d wait for one of the new Ryzen 2400G APUs which would have Ryzen’s under-the-radar ECC support with an IGP (so the mini-ITX PCIe slot could be used for a dedicated storage controller). This would prove to be a big mistake, but more on that in a bit.

2018 NAS

In the end, I did get it all working, but instead of an afternoon of setup, this project dragged on for months and is not quite the traditional NAS I was aiming for. This also just became of of those things where you’d start working on it, run into problems, do some research and maybe order some replacement parts to test out, and then put off because it was such a pain. I figure I’d outline some of the things that went wrong for posterity, as this build was definitely a disaster – the most troublesome build I’ve had in years, if not decades. The problems mostly fell into two categories – SAS, and Ryzen APU (Raven Ridge) issues.

First, the easier part, the SAS issues. This is something that I simply hadn’t encountered, but makes sense looking back. I thought I was pretty familiar w/ how SAS and SATA inter-operated and since I was planning on using a SAS card I had laying around, and using an enclosure w/ a SAS backplane, I figured, what the heck, why not get SAS drives. Well, let me tell you why not – for the HGST Ultrastar He8’s (PDF data sheet) that I bought, the SATA models are SATA III (6Gb/s) models, which will plug into anything and are backwards compatible with just about everything, however the SAS drives are SAS3 (12Gb/s), which it turns out are not backwards compatible at all and require a full SAS3 chain. That means the 6Gb/s (LSI 9207-8i) controller, SFF8087 cables, and the SAS backplane of the otherwise sweet NAS chassis all had to be replaced.

  • I spent time fiddling with applying various firmware updates to the 9207 while I was trying to diagnose my drive errors. This might be relevant for those that decide to go with the 9207 as a SAS controller for regular SATA drives as P20 sometimes has problems vs P19 and you may need to downgrade. You’ll also want the IT FW (these are terrible RAID cards and you want to just pass through the devices). Note, despite the numbering, the 9207 is newer than the 9211 (PCIe 3.0 vs 2.0)
  • Some people have issues with forward vs reverse breakout cables but I didn’t and the spreadsheet at the end of this writeup links to the right cables to buy
  • SAS3 is more expensive across the board – you pay a slight premium (not much) for the drives, and then about double for the controller (I paid $275 for the Microsemi Adaptec 8805E 12Gb/s controller vs $120 for my LSI 9207-8i 6Gb/s controller) and cables (if you’ve made it this far though, $25 is unlikely to break the bank). The biggest pain was finding a 12Gb/s backplane – they didn’t have those for my NAS case, and other cases available were pretty much all ginormous rackmounts. The cheapest option for me ended up being simply buying 2 hot-swap 12Gb/s enclosures (you must get the T3 model w/ the right backplane) and just letting them live free-range
  • BTW, just as a note: if you have a choice between 512b and 4K (AF) sector drives, choose the latter, performance will be much better. If you are using ZFS, be sure to create your pool with ashift=12 to match the sectors

All this work is for bandwidth that honestly, 8 spinning-rust disks are unlikely to use, so if I were doing it over again, I’d probably go with SATA and save myself a lot of time and money.

Speaking of wastes of time and money, the biggest cause of my NAS building woes by far was the Ryzen 2400G APU (Raven Ridge). Quite simply, even as of July 2018, I simply can’t recommend the Ryzen APUs if you’re running Linux. You’ll have to pay more and have slim pickings on the motherboard front if you want mini-ITX and ECC, but you’ll probably spend a lot less time pulling your hair out.

  • I bought the ASRock Mini-ITX board as their reps had confirmed ECC and Raven Ridge support. Of course, the boards in channel didn’t (still don’t depending on manufacture) support the Ryzen APUs out of the box and you can’t boot to update the BIOS without a compatible CPU. AMD has a “boot CPU” program but it was an involved process and after a few emails I just ordered a CPU from Amazon to use and sent it back when I finished (I’ve made 133 Amazon orders in the past 6 months so I don’t feel too bad about that). I had intermittent booting issues (it’d boot a blank screen about half the time) w/ Ubuntu 18.04 until I updated to the latest 4.60 BIOS).
  • With my LSI 9207 card plugged in, 18.04 LTS (4.15 kernel) seemed happy enough (purely with TTYs, I haven’t run any Xorg on this, which has its own even worse set of issues), however with the Adaptec 8805E, it wouldn’t boot at all. Not even the install media would boot on Ubuntu, however, the latest Arch installer would (I’d credit the 4.17 kernel). There’s probably some way to slipstream an updated kernel into LTS installer (my preference generally is to run LTS on servers), but in the end, I couldn’t be that bothered and just went with Arch (and archzfs) on this machine. YOLO!
  • After I got everything seemingly installed and working, I was getting some lockups overnight. These hangs left no messages in dmesg or journalctl logs. Doing a search on Ryzen 2400G, Raven Ridge, and Ryzen motherboard lockups/hangs/crashes will probably quickly make you realize why I won’t recommend Ryzen APUs to anyone. In the end I went into the BIOS and basically disabled anything that might be causing a problem and it seems to be pretty stable (at the cost of constantly high power usage:
    • Disable Cool’n’Quiet
    • Disable Global C-States
    • Disable Sound
    • Disable WAN Radio
    • Disable SATA (my boot drive is NVMe)
    • Disable Suspend to RAM, other ACPI options
  • It’s also worth noting that while most Ryzen motherboards will support ECC for Summit Ridge (Ryzen 1X00) and Pinnacle Ridge (non-APU Ryzen 2X00), they don’t support ECC on Raven Ridge (unbuffered ECC memory will run, but in non-ECC mode), despite Raven Ridge having ECC support in their memory controllers. There’s a lot of confusion on this topic if you do Google searches so it was hard to suss out, but from what I’ve seen, there have been no confirmed reports of ECC on Raven Ridge working on any motherboard. Here’s the way I checked to see if ECC was actually enabled or not:
    # journalctl -b | grep EDAC
    Jul 28 20:11:31 z kernel: EDAC MC: Ver: 3.0.0
    Jul 28 20:11:32 z kernel: EDAC amd64: Node 0: DRAM ECC disabled.
    Jul 28 20:11:32 z kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
    # modprobe -v amd64_edac_mod ecc_enable_override=1
    insmod /lib/modules/4.17.10-1-ARCH/kernel/drivers/edac/amd64_edac_mod.ko.xz ecc_enable_override=1
    modprobe: ERROR: could not insert 'amd64_edac_mod': No such device
    # edac-ctl --status
    edac-ctl: drivers not loaded.
    # edac-ctl --mainboard
    edac-ctl: mainboard: ASRock AB350 Gaming-ITX/ac

OK, so what’s the end result look like? The raw zpool (64TB == 58.2TiB):

# zpool list                                                                                                          
z       58T  5.39T  52.6T         -     0%     9%  1.00x  ONLINE  -    

And here’s what the actual storage looks like (LZ4 compression running):

# df -h | grep z                                                                                                      
Filesystem      Size  Used Avail Use% Mounted on                                                                              
z                40T  4.0T   37T  10% /z

# zfs get compressratio z                                                                                             
NAME  PROPERTY       VALUE  SOURCE                                                                                            
z     compressratio  1.01x  -

Temperatures for the card (I have a 120mm fan pointed at it) and the drives seem pretty stable (+/- 1C or so):

# arcconf GETCONFIG 1 | grep Temperature                                                                              
   Temperature                              : 42 C/ 107 F (Normal)
         Temperature                        : 41 C/ 105 F
         Temperature                        : 44 C/ 111 F
         Temperature                        : 44 C/ 111 F
         Temperature                        : 40 C/ 104 F
         Temperature                        : 40 C/ 104 F
         Temperature                        : 43 C/ 109 F
         Temperature                        : 43 C/ 109 F
         Temperature                        : 40 C/ 104 F

Performance seems decent, about 300MB/s copying from a USB-C/SATA SSD. Here is are the results of an iozone (3.482) run (settings taken from this benchmark):

# iozone -i 0 -i 1 -t 1 -s16g -r16k  -t 20 /z                                   
        File size set to 16777216 kB
        Record Size 16 kB
        Command line used: iozone -i 0 -i 1 -t 1 -s16g -r16k -t 20 /z                  
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.                                       
        Processor cache line size set to 32 bytes.                                     
        File stride size set to 17 * record size.                                      
        Throughput test with 20 processes
        Each process writes a 16777216 kByte file in 16 kByte records                  

        Children see throughput for  1 initial writers  = 1468291.75 kB/sec            
        Parent sees throughput for  1 initial writers   = 1428999.28 kB/sec            
        Min throughput per process                      = 1468291.75 kB/sec            
        Max throughput per process                      = 1468291.75 kB/sec            
        Avg throughput per process                      = 1468291.75 kB/sec            
        Min xfer                                        = 16777216.00 kB               

        Children see throughput for  1 rewriters        = 1571411.62 kB/sec            
        Parent sees throughput for  1 rewriters         = 1426592.78 kB/sec            
        Min throughput per process                      = 1571411.62 kB/sec            
        Max throughput per process                      = 1571411.62 kB/sec            
        Avg throughput per process                      = 1571411.62 kB/sec            
        Min xfer                                        = 16777216.00 kB               

        Children see throughput for  1 readers          = 3732752.00 kB/sec            
        Parent sees throughput for  1 readers           = 3732368.39 kB/sec            
        Min throughput per process                      = 3732752.00 kB/sec            
        Max throughput per process                      = 3732752.00 kB/sec            
        Avg throughput per process                      = 3732752.00 kB/sec            
        Min xfer                                        = 16777216.00 kB               

        Children see throughput for 1 re-readers        = 3738624.75 kB/sec            
        Parent sees throughput for 1 re-readers         = 3738249.69 kB/sec            
        Min throughput per process                      = 3738624.75 kB/sec            
        Max throughput per process                      = 3738624.75 kB/sec            
        Avg throughput per process                      = 3738624.75 kB/sec            
        Min xfer                                        = 16777216.00 kB               

        Each process writes a 16777216 kByte file in 16 kByte records                  

        Children see throughput for 20 initial writers  = 1402434.54 kB/sec            
        Parent sees throughput for 20 initial writers   = 1269383.28 kB/sec            
        Min throughput per process                      =   66824.69 kB/sec            
        Max throughput per process                      =   73967.23 kB/sec            
        Avg throughput per process                      =   70121.73 kB/sec            
        Min xfer                                        = 15157264.00 kB               

        Children see throughput for 20 rewriters        =  337542.41 kB/sec            
        Parent sees throughput for 20 rewriters         =  336665.90 kB/sec            
        Min throughput per process                      =   16713.62 kB/sec            
        Max throughput per process                      =   17004.56 kB/sec            
        Avg throughput per process                      =   16877.12 kB/sec            
        Min xfer                                        = 16490176.00 kB     

        Children see throughput for 20 readers          = 3451576.27 kB/sec            
        Parent sees throughput for 20 readers           = 3451388.13 kB/sec            
        Min throughput per process                      =  171099.14 kB/sec            
        Max throughput per process                      =  173923.14 kB/sec            
        Avg throughput per process                      =  172578.81 kB/sec            
        Min xfer                                        = 16505216.00 kB               

        Children see throughput for 20 re-readers       = 3494448.80 kB/sec            
        Parent sees throughput for 20 re-readers        = 3494333.50 kB/sec            
        Min throughput per process                      =  173403.55 kB/sec            
        Max throughput per process                      =  176221.58 kB/sec            
        Avg throughput per process                      =  174722.44 kB/sec            
        Min xfer                                        = 16508928.00 kB               

While running this it looked like the each of the 4 cores hit about 10-15% on htop and load about 21-22 (waiting on iozone of course). Here are the arcstats during the top of the run:

# 1
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c              
18:15:40  266K   24K      9     0    0   24K  100    24    0   7.2G  7.2G              
18:15:41  293K   26K      9     2    0   26K  100    28    0   7.1G  7.1G              
18:15:42  305K   27K      9     0    0   27K  100    26    0   7.1G  7.1G              

Anyway, those were some surprising big (totally synthetic numbers), but I don’t have much of a reference, so a comparison, I ran the same test on the cheap ADATA M.2 NVMe SSD that I use for my boot drive:

# iozone -i 0 -i 1 -t 1 -s16g -r16k -t 20 ./
	File size set to 16777216 kB
	Record Size 16 kB
	Command line used: iozone -i 0 -i 1 -t 1 -s16g -r16k -t 20 ./
	Output is in kBytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 kBytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
	Throughput test with 20 processes
	Each process writes a 16777216 kByte file in 16 kByte records

	Children see throughput for  1 initial writers 	=  737763.19 kB/sec
	Parent sees throughput for  1 initial writers 	=  628783.43 kB/sec
	Min throughput per process 			=  737763.19 kB/sec 
	Max throughput per process 			=  737763.19 kB/sec
	Avg throughput per process 			=  737763.19 kB/sec
	Min xfer 					= 16777216.00 kB

	Children see throughput for  1 rewriters 	=  537308.31 kB/sec
	Parent sees throughput for  1 rewriters 	=  453965.01 kB/sec
	Min throughput per process 			=  537308.31 kB/sec 
	Max throughput per process 			=  537308.31 kB/sec
	Avg throughput per process 			=  537308.31 kB/sec
	Min xfer 					= 16777216.00 kB

	Children see throughput for  1 readers 		=  710123.75 kB/sec
	Parent sees throughput for  1 readers 		=  710108.56 kB/sec
	Min throughput per process 			=  710123.75 kB/sec 
	Max throughput per process 			=  710123.75 kB/sec
	Avg throughput per process 			=  710123.75 kB/sec
	Min xfer 					= 16777216.00 kB

	Children see throughput for 1 re-readers 	=  709986.50 kB/sec
	Parent sees throughput for 1 re-readers 	=  709970.87 kB/sec
	Min throughput per process 			=  709986.50 kB/sec 
	Max throughput per process 			=  709986.50 kB/sec
	Avg throughput per process 			=  709986.50 kB/sec
	Min xfer 					= 16777216.00 kB

# oops, runs out of space trying to run the 20 thread test
# only 90GB free on the boot drive...

One more iozone test, pulled from this benchmark. The ZFS volume:

	Using minimum file size of 131072 kilobytes.
	Using maximum file size of 1048576 kilobytes.
	Record Size 16 kB
	OPS Mode. Output is in operations per second.
	Auto Mode
	Command line used: iozone -n 128M -g 1G -r 16 -O -a C 1
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 kBytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          131072      16   107383   153753   454176   497752   402551   100747   337201    228240    143842   149257   107891   243222   227732
          262144      16   114165   134997   194843   209168    62298    78727   166649    227852     47554   120907   121755   206258   208830
          524288      16   108586   130493   228032   235020    53501    48555   190495    224892     45273   110338   113536   229965   205326
         1048576      16    83337    94119   203190   231459    46765    34392   180697    230120     44962    92476   112578   198107   230100

And the boot NVMe SSD:

	Using minimum file size of 131072 kilobytes.
	Using maximum file size of 1048576 kilobytes.
	Record Size 16 kB
	OPS Mode. Output is in operations per second.
	Auto Mode
	Command line used: iozone -n 128M -g 1G -r 16 -O -a C 1
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 kBytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          131072      16   124072   218418   407623   392939   462018   211145   392298    520162    435927   218552   206769   411371   453720
          262144      16   125471   236933   454936   427993   449000   212884   423337    525110    452045   229310   221575   451959   494413
          524288      16   123998   252096   520458   459482   511823   229332   496952    526485    509769   243921   239714   519689   547162
         1048576      16   125236   266330   562313   480948   476196   220034   498221    529250    471102   249651   247500   560203   571394

And then one more quick comparison testing using bonnie++. The ZFS volume:

# bonnie++ -u root -r 1024 -s 16384 -d /z -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
z               16G           1084521  82 933471  88           2601777  99  1757  52
Latency                         103ms     103ms              3469us   92404us
Version  1.97       ------Sequential Create------ --------Random Create--------
z                   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1   125   1 +++++ +++   117   1   124   1 +++++ +++   120   1
Latency               111ms       4us   70117us   49036us       5us   43939us

And the boot NVMe SSD:

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
z               16G           118358   7 49294   3           392088  10  5793  63
Latency                         178ms    1121ms              1057ms   32875us
Version  1.97       ------Sequential Create------ --------Random Create--------
z                   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1  2042   3 +++++ +++   857   1   767   1 +++++ +++   839   1
Latency              2135us     106us   39244us   10252us       5us   35507us

Now granted, this is a cheap/slow NVMe SSD (I have a 512GB 970 Pro in a box here, but I’m too lazy/don’t care enough to reinstall on that to test), but the ZFS results surprised me. Makes you wonder whether an array of enterprise SAS SSDs would beat out say those PCIe SSD cards, but I don’t get revved enough about storage speeds to really do more than pose the question. I may do a bit more reading on tuning, but I’m basically limited by my USB and network connection (a single Intel I211A 1 Gbps) anyways. Next steps will be centralizing all my data, indexing, deduping, and making sure that I have all my backups sorted. (I may have some files that aren’t’ backed up, but that’s outweighed by many, many files that I probably have 3-4 copies of…)

Oh, and for those looking to build something like this (again, I’d reiterate: don’t buy a Ryzen APU if you plan on running Linux and value your sanity), here’s the final worksheet that includes the replaced parts that I bought putting this little monster together (interesting note: $/GB did not go down for my storage builds for the past few years):

Misc notes:

  • If you boot with a USB attached it’ll boot up mapped as /dev/sda, nbd if you’re mapping your ZFS properly
  • Bootup takes about 1 minute – about 45s of that is with the controller card BIOS
  • I replaced the AMD stock cooler w/ an NH-L9a so it could fit into the NSC-810 NAS enclosure, but obviously, that isn’t needed if you’re just going to leave your parts out in the open (I use nylon M3 spacers and shelf liners to keep from shorting anything out since I deal with a lot of bare hardware these days)

Printing with a DYMO LabelManager PNP on Linux

A while back I bought a DYMO LabelManager Plug N Play Label Maker to help organize things. Before I bought it I had done some cursory searching and saw there were Linux drivers, and I set up CUPS, fixed permissions (of course), and installed the DYMO drivers, but nothing was showing up.

Well, turns out there’s a reason for that – the LabelManager PnP actually labels itself as a HID device, not a printer! (lsusb -v to peep the details)…

Luckily, with a bit of searching, I found a nice little Python 3 script called dymoprint (github) that reverse-engineered the USB protocol and works perfectly. Another dev subsequently wrote a Perl script that generates 64px tall bitmaps to the printer. (I have lots of existing image generation code to build a Python version of this, but honestly, the first dymoprint script does just about everything I want, which is just to print some simple labels).

I saw that someone else did get it to run on CUPS – it looks like you have to set up modeswitching, but I haven’t tested that personally.

Switching from urxvt to termite

I’ve been running urxvt for years and have it pretty much dialed in as I like it with base16 Ocean and a bunch of mostly font-specific options. It works reliably and quickly for me, without any muss or fuss (I run my Openbox without any menubars and the chromeless, super minimal look fits right in).

There’s just one problem. I use nload a lot (bandwidth monitoring tools deserve it’s own writeup one day), and it flickers in urxvt. Now, to be clear, this isn’t a bug with urxvt, which is merely doing what it’s told, but I noticed that this sort of ncurses flicker on clear screen, due to the way it buffers, doesn’t happen in vte-based terminal emulators.

I’d previously tried a few other terminal emulators (Terminator, sakura) and I gave alacritty and xst and (a few others) a good try this time around, but I ended up settling on termite as basically a drop-in replacement for urxvt.

Setting up the fonts and colors was relatively painless and you can even dynamically reload those. The one niggle was that the left scrollbar was much more distracting than in urxvt. A little bit of searching led me to a related issue, which pointed me on ways to update GTK styles. It took a bit of tweaking to get things just right (there was some stray corner rounding that required some creative CSS), but here’s where I ended up with my ~/.config/gtk-3.0/gtk.css:

.termite scrollbar.vertical slider {
  background-color: #969896;
  border: 0;
  min-width: 7px;
  border-radius: 0;

.termite scrollbar.vertical {
  background: none;
  margin: 0;
  padding: 0;
  padding-right: 2px;

  border: 0;
  border-right: 1px solid rgba(0,0,0,0);
  border-radius: 0;

One thing to note, is that termite is actually a bit slower than urxvt (even ignoring it’s slightly weird refresh – when showing lots of text it tends to buffer and seems to skip rendering things you might see zip by in other terminal emulators), but it does handle mpv --vo tct rendering correctly (whereas my urxvt just barfs). For some more on terminal emulator performance, this alacritty github issue is a good start (alacritty may be hot shit on OS X, but it’s slower than urxvt on Linux and I don’t like its HiDPI handling). Also related and interesting is this Dan Luu writeup on Terminal latency.

And that wraps up today’s episode of Yak Shavers. This might become an ongoing series as I tweak some of the remaining issues on my Linux systems (next might be migrating from 1Password due to the Firefox plugin being broken, or better notification supression when videos are playing). Past issues/fixes have been largely chronicled in my Arch Linux Install doc, although a number of new things are in a private wiki. One of my goals this year is to figure out the best way to publish most of that stuff publicly.

WordPress Security

By far the most insecure piece of software that I still run on my main web server these days (where you’re reading this!) is WordPress. It seems like there’s never more than a few months (also) that go by without some new XML-RPC exploit or some-such pops up. The easiest way to stay reasonably secure is with regular updates. About 4 years ago I automated that with a simple daily WP-CLI (best tool) update script that basically looks like:

wp core update
wp plugin update --all
wp theme update --all

There look like there are some slightly better featured scripts (also also) floating around, but it seems to do the job.

I also run a few security plugins, like Activity Log, WP fail2ban, and Sucuri Security and I haven’t seemed to have had too many problems over the past few years on my main blog, however my terribly neglected travel blog apparently wasn’t getting regular updates this past year and needed a bit of delousing (some spam urls etc, that just needed to be reverted) – the sad thing is that it had an update script, but wasn’t being run in cron (wah wah).

2017 Year In Review

I originally had somewhat more ambitious plans for my 2017 wrap up, but well, the end of the year is just about here so instead I’ll just type for a couple hours, hit publish, and call it a day.

Part of the motivation is that it’s felt like a good time again to write up some of what I’ve been thinking about in technology trends. In 2006, while I was hip-deep in Web 2.0 work (and my blog output had already fallen into the abyss where it remains today) and I wrote up a 5 year tech projection. I ended up revisiting it 5 years later and you know what, didn’t too badly. What’s interesting reviewing it now is the a few of the things that I had missed were actually on the cusp then and happened shortly after. I didn’t do a direct followup, but did do a 2013 Review in Tech writeup – the most interesting things that happened that year weren’t in consumer/SV tech scene (which was deep in their Uber for X/app obsession at the time).

In 2014 I started collecting some Emerging Tech notes that I never published. That might be worth checking out (there are some late 2017 notes as well) – these seemed to have caught the tech zeitgeist a couple years in advance but it’s a bit fuzzy on how these will play out. This year, I also started collected some notes on a future-trend focused Tumblr (it’s not private per-se, just not very publicized/widely read, although the same can be said for this blog at this point – just pissing into the wind). For 2018, I’m hoping to both publish more and to better rationalize where/how I’m publishing what I’m tracking.

In many ways 2017 was a trash fire, so before I dig into it, I did want to start off Charlie Stross Reasons to Be Cheerful style. In just about every measure of human development and global, now is the best time to be alive. For those that are interested in visualizations, the World Bank’s 2017 Atlas of Sustainable Development Goals is simply fantastic (Hans Rosling style, RIP) and again, shows the marked improvements we’ve made. If you’re into listicles, this recent article 99 Reasons 2017 Was A Great Year highlights some nice things. I started a Twitter list today that I hope to add to dedicated to accounts focused on sharing useful metrics/trends on the state of the world.

Now enough of that, and into the weeds. Per usual, I spent a lot of time reading things this year (example) – too much on Twitter and Reddit, but on the whole, more worthwhile things than not – I spent a fair amount of time digging through writings of the socio-techno-political variety, lots on crypto-economics and other financial topics, and rounded off by the usual geek topics. Also, a lot more YouTube than usual. This marked year 4 of semi-nomadicism although I may spend some more time settled to try to get through a backlog of housekeeping. Being out and about in different parts of the world helps give some perspective (places visited for the first time included Colombia, Cuba, Iceland, Greece, Kazakhstan, and Brazil).

Like many others, I spent much of the end of last year and the beginning of this year reading and thinking about the state (and fate) of liberal democracy in the modern world. I collected some of that into a doc Sensemaking in the Age of Social Media. While most of the participants haven’t realized it yet (or are disingenously denying it), we are now living in the age of weaponized information – memetic warfare. This is as cyberpunk and dystopian as it sounds, and it’s worth giving a shout out to sci-fi authors. The easiest way to understand where we are is to re-read Gibson, Sterling, Stephenson, Egan, Stross, Doctorow et al with the lens of what we are experiencing. It’s also worth thinking about how unprepared humans and human societies currently are against the future-shock mechanization of the modern infosphere (hyper-personalization and filter bubbles, bot/troll manipulation and other social signal hacks, infoglut and overload, clickbait and yes, fake news). These are second order effects that web pioneers and SV techies were unprepared for and misincentivized to address (who knew that driving engagement for advertising revenue would bring down free society, wah wah). This of course made it’s way into the news zeitgeist this year (that the modern media landscape is a key part of this dysfunction is an irony that is sadly lost to most, I believe). A smattering of headlines: Former Facebook executive: social media is ripping society apart, Facebook must wake up to its disastrous potential – it has the power to subvert American democracy, What Facebook Did to American Democracy, Facebook Wins, Democracy Loses, Can democracy survive Facebook? – now this is all a bit unfair to Facebook, after all Twitter is perhaps even more of a trash fire (and @realDonaldTrump will probably start WW3 on it next year). Anyway, before I go full rant – there aren’t easy answers, but it’s clear that we must fix this. These are design failures – some driven purposefully by misaligned economic incentives and externalized risk, and some by the short-sightedness and failings of designers, engineers, and product managers. IMO, if we can not fix this, humanity will probably not survive.

Over the course of the year I tried to crystallize a line of thought – that there were no problems humanity faced that could not be solved, if we could solve the problem of how to cooperate in rational self interest. Not such a deep insight, and not pithy enough yet (still a work in progress, obviously) but good enough as a direction to point one’s mental energy and efforts towards. (For those in doubt, and as a benchmark for this, nominal global GDP is about 80T USD – look at any looming existential crisis that we face and ask how much actual effort/cost it would take to address, mitigate, or fix.)

Also tying into perhaps the next topic, on cryptocurrencies. Or perhaps, more accurately a discussion on distributed trust network, or resilient distributed consensus in the presence of byzantine adversaries, or about censorship-resistant transactions, or incentivization structures for said networks.

Yes, we are currently in a bit of a mania phase of a bubble at the moment. One that hasn’t, but will inevitably pop (although I wouldn’t pack it in until the institutional money gets a dip – this might not even be the big bubble yet in the same way that 2014 wasn’t). At the end of it though we’ll be where we were at the end of the Internet bubble – with a whole bunch of new toys to play with that with the power to reshape society. Hopefully, having gone through it once already, we can try again a bit wiser.

A few interesting recent reads that might spark some ideas:

The Blockchain Economy: A beginner’s guide to institutional cryptoeconomics

The great ICO freeze is beginning. What does it mean for crypto in 2018?

Co-evolving the Phase Shift to Crypto Capitalism by Founding The Ethereum Commons Co-op

OK, this is getting long, and there are so many other things I want to cover.

I’m a big fan of Ramez Naam’s talks and projection on energy production costs and trends. Here’s a recent one from October:

I’ve previously linked to a fascinating writeup on AlphaZero and DeepMind. Here’s a Year in Review of AI and Deep Learning in 2017.

I’ve been thinking a fair bit about the “4chan problem” – there’s the whole Gamergate to literal nazis thing, but just repeated over, and over, the emergence (and maybe this is the actual new part, the ability to cause damage) of a generation of sociopathic man-children. The Mirai botnet came out of some socially maladjusted Minecrafters (a fascinating read; followup, background, interesting related color on how fucked IoT security is) and say, the recent swatting incident (follow @briankrebs for fascinating infosec/cybercrime insights, @radleybalko for reporting on police brutality, militarization, and how messed up criminal justice and civil liberties are in the US).

OK, well, enough of that. Perhaps a bit less on the tech insights than a more planned essay would have been. My resolution for the coming year will be to figure out a better way of collecting and publishing my research on an ongoing basis. Maybe not quite gwern style but I think that a lot of what I come across and read about might be useful to others, and the act of publishing would probably encourage better organization/clear thinking. Another resolution: trying to waste less time on the Internet.

Feels Like the Future

I’ve been gathering up some year-end thoughts, but in the meantime, some things I’ve stumbled upon that point to some trends I’ve been paying attention to…

In more pedestrian grim meathookness, Nilay Patel has a good summary on the net neutrality repeal: Ajit Pai just handed Republicans a bag of shit

Busy Bee

One of the things that never fails to surprise me is the sheer amount of amazing/interesting stuff that pops up every day. Over the past few months, I’ve quietly been trying to capture a few of the highlights and have been planning on figuring out a better system, here for example is a more complete list of stuff just from the past day or two of my reading. Most of this I have just sent to Pocket or my ever growing Watch List (there simply aren’t enough hours in the day):

It’s worth noting that this is a sampling from about one day’s worth of bookmarks one three aggregators (Twitter, Hacker News, and Reddit) that I only check a few times over the course of the day (this was on a travel day no less). Also that simply going back and gathering and sifting all these out took about an hour this morning.

Putting a little bit of thought on how to better manage all this.

Ryzen Build

With the 4K drone footage I’ve been editing, I’ve been really pushing my Gigabyte Aero 14 past its limits, and since I was planning on spending a fair bit of time in one place for the next couple months (Tokyo until June), and since my bet on AMD last year has paid off well, it seemed fitting to put together a new Ryzen system.

I was traveling most of March, so I wasn’t in a launch-day rush, but after reading some reviews, I saved a couple hundred bucks and went with a $320 Ryzen 7 1700. As a bonus, the 1700 comes with a nice looking (and pretty functional) CPU cooler and my copy easily overclocks to 3.7GHz (3.8-3.9 pushing voltages, but I’d probably upgrade the cooler in that case – you can purchase pre-binned versions here).

RAM prices have gone through the roof the past few months (almost double the price over the past 6mo) – I bought a pair of TridentZ 3200/CL16 sticks (Samsung E-Die) that I’m able to currently run at 2933 on my motherboard, but apparently, if you’re looking for max performance, you’ll want to go for some Samsung B-Die RAM at the moment.

I bought an MSI X370 Gaming Pro Carbon primarily because it was a nice monochrome look and had dual M.2 support (only the first slot runs at PCIe 3.0 x4 sadly due to the Ryzen 7/x370’s PCIe lane availability), but I wouldn’t recommend it. While it’s had some BIOS updates, it still doesn’t have the latest 4/10 AGESA update, and in general has had sluggish support and a few issues (my personal gripes: pokey POSTing, no last-good/soft-CMOS reset). Personally, if I was buying a top-of-the-line board, I’d probably go with the Asus ROG CH6 – while it doesn’t have a second M.2 slot, you could put a PCIe adapter board on the last slot for that. As a bonus, there are a bajillion USB ports.

I’m running a Corsair SF600 SFX Power Supply (9.7 jonnyGURU Recommended) on an Open Benchtable BC1 which works/looks great, w/ the caveat that SFX power supply requires a 3D-printed adapter to mount.

Instead of buying new GPUs, I just brought along a couple RX470 mining cards (sadly, these two weren’t running the past two months – that would have been $400-600 of missed earnings w/ the ETH run-up). I have them beavering away in the background right now while the system idles. (I am running the latest 17.4.2 drivers but with a BIOS signature check bypass).

One note during installation is that I had hard-lockup problems when installing from a 2015 Windows 10 stick – you’ll want to make a new one, w/ a 12/2016 ISO I didn’t have any problems.

Alienware 13R3 Review

While my VR development has been a bit sidetracked recently (one guess why), with the release of Nvidia’s Pascal 10-series mobile GPUs, I’ve been looking forward to abandoning my previous portable VR workstations and switching to a more traditional, relatively compact laptop.

I was originally most interested in the Aorus X3 Plus V6, but its release was so delayed, that I ended up preordering the Dell Alienware 13R3 when it was announced in November. Due to some discount hijinks (and the fact that it hadn’t shipped when it was scheduled to), I ended up reordering during the Black Friday sales. Delivery was originally scheduled for Dec 5, but got pushed back until the 22nd, by which time I was already out of the country, so I only this week finally had a chance to break it out and put it through its paces.

So, first, some positive things:

  • OLED screen – the main reason I decided I wanted the Alienware 13, despite the long wait and a few other concerns, was because of the screen. While only 60Hz, the OLED pixels really are glorious – sharp, contrasty, with 1ms switching and a ridiculous color gamut (104% AdobeRGB). It’s a capacitive multitouch screen (I don’t like to touch my screens but the one time I accidentally did, it seemed to work), and it’s mounted on a very solid hinge that doesn’t not wobble *at all* when typing. There is an achilles heel, but I’ll save that for the ‘cons’ section.
  • Good performance – If you order the 13R3 now, you’ll get the latest Kaby Lake (7-series) processor – mine was in the one-month window where the Skylake (6-series) shipped, however the performance difference is minimal (the Kaby Lake might have an ever so slightly better boost clock). I got mine with the highest specced i7-6700HQ that despite worries due to thermal problems in many early subreddit threads, ended up running fine. CPU stress tests clocked it at about 75-80C running at a boost clock of 3.1-3.2GHz, with even lower temperatures undervolting w/ Intel XTU. Sadly, when running benchmarks like Valley or gaming, the system still ended up being about 20% slower than my compact (stock) i7-4790K and GTX970 combo, but I doubt that any other GTX1060 based portable system would do much better (the GPU clock of the GTX1060 didn’t boost to 1900MHz, but stayed a reasonable 1600-1700MHz at around 80-85C while gaming).
  • The keyboard and trackpad were top notch, both in feel, and surprisingly, with the lighting on the trackpad (this is configurable, but by default the trackpad glows when in use and it’s actually pretty neat). The top plate is a soft touch plastic that is nice, although even over a couple days of light use, has started to acquire a few spots of sheen (ick, I know).
  • The Webcam has IR support for Tobii eye tracking (never used) and Windows Hello logins (which actually works great and is delightful)
  • Build quality is super solid – it’s built like a tank and it feels like you could definitely use it as a bludgeoning tool and then continue on your merry way. It’s also worth noting that swapping RAM (2 slots) or m.2 SSDs (2 slots) is a breeze – literally 5 phillips screws on the bottom, which is a big plus. The service manual is online, and overall, near complete disassembly looks like a breeze.
  • Due to the delays, I ended up calling Dell customer service a few times, and while not always completely helpful, I didn’t have to spend too long on hold and most of the time it felt like they were moving the ball along, so kudos for that.

OK, now with the cons, which includes some pretty serious stuff, sadly.

  • The screen – As I mentioned, one of the main reasons I picked the Alienware 13 over anything else was the OLED screen. And it really is glorious – as long as you are in complete darkness that is (that Achilles heel I mentioned). Even in indirect or low ambient light, the screen is basically a mirror, and if you like running dark text terminals like me (which would also be better for the OLED’s battery life), you will spend a lot of time staring at your own reflection. I really can’t fathom why someone would have such an otherwise awesome display and then put it behind such a glossy piece of glass.
  • Size and weight – While the build quality is admirable, and in theory I knew it was heavier than some of the alternatives, it’s not until you get it in your hands do you realize how bulky it really is. The computer itself is 2.5kg (5.5lb), and the power brick is another 0.8kg (1.7lb) on top of that. In total, you’re looking at almost 3.3kg (7.2lb) for the package. The sizing isn’t much better. It’s relatively thick at about 24-27mm (the other dimensions aren’t super small either), but worst of all, the center of the laptop is actually pointy, not flat. This makes propping the laptop up or using many laptop stands a non-starter. I don’t really know what Alienware was thinking with that design element.
  • Battery life – shorter battery life is something that I expected but in practice, turned out to be unacceptably low (much lower than reviews and claims I had seen) 84t92ao. In my unscientific rundown test of random web browsing and YouTube video watching, I got 3h 10m of use from a full charge before it shut down. This was in a darkened room and I don’t think I heard the fans turn on once mind you, so I don’t think it was stressing the system (also, w/ the OLED screen, you can’t disable Optimus, so I assume it wasn’t working out the dGPU even).
  • I paid a few bucks extra for the Killer 1535 (vs 1435) network card, which is supposed to have solved some of the older 1435’s connection problems. I also uninstalled the Killer Suite and reinstalled just the drivers, as that’s supposed to help as well. Alas, for whatever reason, the 1535 would drop connection (well, remain connected but time out on packets) about once a day. I have half a dozen devices running on my AC wifi without issues, so I’ll lay the blame on the card – it’s a relatively minor issue since it’d only be about $30 and 5 minutes to swap it out w/ an Intel wifi card, but I figure I’d mention it, while I’m piling on.
  • As I mentioned, there’s some minor throttling under gaming loads (1600-1700MHz on the GPU vs the 1900MHz max boost), although angling the laptop for better ventilation didn’t seem to improve things much. I also noticed some minor occassional graphics memory corruption in certain overlays in the game. Note: performance improved by about 10% after updating to the latest Nvidia WHQL drivers. There’s probably more I could have done to tweak out performance, but the screen, life, and bulk really killed my enthusiam

As you might expect, after a few days realizing some of the shortcomings, I’ve ended up deciding to return the system. I think for those that don’t fly/travel as much, or that don’t mind extremely glossy screens (there must be a lot of people like that, because I feel like Dell isn’t the only offender here), this might be a good fit. There really are a lot to admire here, and the reviews that I read/watched were mostly positive, so I don’t want to give the impression that this laptop is a total stinker.

For those looking at lightweight/portable VR capable workstations, your options are still sort of limited. I’ve ordered a Gigabyte Aero 14 that will hopefully address the worst issues I had with the Alienware 13 – it has a matte anti-glare IPS display (sadly w/ average color gamut), is 1.9kg (the 2.4kg weight with the power adapter is the same as the weight of the Alienware laptop by itself) and is almost 50% thinner. It also has a beefier 94Wh battery (Gigabyte has made claims for 10 hours of use, but honestly, I’d be happy if it could hit 5h of web browsing) and an Intel 8260 wifi card. Gentech is also offering a free liquid metal CPU+GPU repaste so I have some hopes that performance might actually be a bit better as well.

Here’s a couple video reviews of the Alienware 13:

(Just look at that glare in the Linus video – he doesn’t mention it at all)

Oh, and here’s some Aero 14 Reviews