By far the most insecure piece of software that I still run on my main web server these days (where you’re reading this!) is WordPress. It seems like there’s never more than a few months (also) that go by without some new XML-RPC exploit or some-such pops up. The easiest way to stay reasonably secure is with regular updates. About 4 years ago I automated that with a simple daily WP-CLI (best tool) update script that basically looks like:
I also run a few security plugins, like Activity Log, WP fail2ban, and Sucuri Security and I haven’t seemed to have had too many problems over the past few years on my main blog, however my terribly neglected travel blog apparently wasn’t getting regular updates this past year and needed a bit of delousing (some spam urls etc, that just needed to be reverted) – the sad thing is that it had an update script, but wasn’t being run in cron (wah wah).
I originally had somewhat more ambitious plans for my 2017 wrap up, but well, the end of the year is just about here so instead I’ll just type for a couple hours, hit publish, and call it a day.
Part of the motivation is that it’s felt like a good time again to write up some of what I’ve been thinking about in technology trends. In 2006, while I was hip-deep in Web 2.0 work (and my blog output had already fallen into the abyss where it remains today) and I wrote up a 5 year tech projection. I ended up revisiting it 5 years later and you know what, didn’t too badly. What’s interesting reviewing it now is the a few of the things that I had missed were actually on the cusp then and happened shortly after. I didn’t do a direct followup, but did do a 2013 Review in Tech writeup – the most interesting things that happened that year weren’t in consumer/SV tech scene (which was deep in their Uber for X/app obsession at the time).
In 2014 I started collecting some Emerging Tech notes that I never published. That might be worth checking out (there are some late 2017 notes as well) – these seemed to have caught the tech zeitgeist a couple years in advance but it’s a bit fuzzy on how these will play out. This year, I also started collected some notes on a future-trend focused Tumblr (it’s not private per-se, just not very publicized/widely read, although the same can be said for this blog at this point – just pissing into the wind). For 2018, I’m hoping to both publish more and to better rationalize where/how I’m publishing what I’m tracking.
Now enough of that, and into the weeds. Per usual, I spent a lot of time reading things this year (example) – too much on Twitter and Reddit, but on the whole, more worthwhile things than not – I spent a fair amount of time digging through writings of the socio-techno-political variety, lots on crypto-economics and other financial topics, and rounded off by the usual geek topics. Also, a lot more YouTube than usual. This marked year 4 of semi-nomadicism although I may spend some more time settled to try to get through a backlog of housekeeping. Being out and about in different parts of the world helps give some perspective (places visited for the first time included Colombia, Cuba, Iceland, Greece, Kazakhstan, and Brazil).
Like many others, I spent much of the end of last year and the beginning of this year reading and thinking about the state (and fate) of liberal democracy in the modern world. I collected some of that into a doc Sensemaking in the Age of Social Media. While most of the participants haven’t realized it yet (or are disingenously denying it), we are now living in the age of weaponized information – memetic warfare. This is as cyberpunk and dystopian as it sounds, and it’s worth giving a shout out to sci-fi authors. The easiest way to understand where we are is to re-read Gibson, Sterling, Stephenson, Egan, Stross, Doctorow et al with the lens of what we are experiencing. It’s also worth thinking about how unprepared humans and human societies currently are against the future-shock mechanization of the modern infosphere (hyper-personalization and filter bubbles, bot/troll manipulation and other social signal hacks, infoglut and overload, clickbait and yes, fake news). These are second order effects that web pioneers and SV techies were unprepared for and misincentivized to address (who knew that driving engagement for advertising revenue would bring down free society, wah wah). This of course made it’s way into the news zeitgeist this year (that the modern media landscape is a key part of this dysfunction is an irony that is sadly lost to most, I believe). A smattering of headlines: Former Facebook executive: social media is ripping society apart, Facebook must wake up to its disastrous potential – it has the power to subvert American democracy, What Facebook Did to American Democracy, Facebook Wins, Democracy Loses, Can democracy survive Facebook? – now this is all a bit unfair to Facebook, after all Twitter is perhaps even more of a trash fire (and @realDonaldTrump will probably start WW3 on it next year). Anyway, before I go full rant – there aren’t easy answers, but it’s clear that we must fix this. These are design failures – some driven purposefully by misaligned economic incentives and externalized risk, and some by the short-sightedness and failings of designers, engineers, and product managers. IMO, if we can not fix this, humanity will probably not survive.
Over the course of the year I tried to crystallize a line of thought – that there were no problems humanity faced that could not be solved, if we could solve the problem of how to cooperate in rational self interest. Not such a deep insight, and not pithy enough yet (still a work in progress, obviously) but good enough as a direction to point one’s mental energy and efforts towards. (For those in doubt, and as a benchmark for this, nominal global GDP is about 80T USD – look at any looming existential crisis that we face and ask how much actual effort/cost it would take to address, mitigate, or fix.)
Also tying into perhaps the next topic, on cryptocurrencies. Or perhaps, more accurately a discussion on distributed trust network, or resilient distributed consensus in the presence of byzantine adversaries, or about censorship-resistant transactions, or incentivization structures for said networks.
Yes, we are currently in a bit of a mania phase of a bubble at the moment. One that hasn’t, but will inevitably pop (although I wouldn’t pack it in until the institutional money gets a dip – this might not even be the big bubble yet in the same way that 2014 wasn’t). At the end of it though we’ll be where we were at the end of the Internet bubble – with a whole bunch of new toys to play with that with the power to reshape society. Hopefully, having gone through it once already, we can try again a bit wiser.
A few interesting recent reads that might spark some ideas:
OK, well, enough of that. Perhaps a bit less on the tech insights than a more planned essay would have been. My resolution for the coming year will be to figure out a better way of collecting and publishing my research on an ongoing basis. Maybe not quite gwern style but I think that a lot of what I come across and read about might be useful to others, and the act of publishing would probably encourage better organization/clear thinking. Another resolution: trying to waste less time on the Internet.
One last cryptocurrency and society link, this essay on ledgers and “cryptoeconomics” (defined within as “the institutional consequences of cryptographically secure and trustless ledgers is some good food for thought.
One of the things that never fails to surprise me is the sheer amount of amazing/interesting stuff that pops up every day. Over the past few months, I’ve quietly been trying to capture a few of the highlights and have been planning on figuring out a better system, here for example is a more complete list of stuff just from the past day or two of my reading. Most of this I have just sent to Pocket or my ever growing Watch List (there simply aren’t enough hours in the day):
The Shouting Class – a fantastic and very insightful essay on how social media has changed the nature of social discourse
It’s worth noting that this is a sampling from about one day’s worth of bookmarks one three aggregators (Twitter, Hacker News, and Reddit) that I only check a few times over the course of the day (this was on a travel day no less). Also that simply going back and gathering and sifting all these out took about an hour this morning.
Putting a little bit of thought on how to better manage all this.
I was traveling most of March, so I wasn’t in a launch-day rush, but after reading some reviews, I saved a couple hundred bucks and went with a $320 Ryzen 7 1700. As a bonus, the 1700 comes with a nice looking (and pretty functional) CPU cooler and my copy easily overclocks to 3.7GHz (3.8-3.9 pushing voltages, but I’d probably upgrade the cooler in that case – you can purchase pre-binned versions here).
I bought an MSI X370 Gaming Pro Carbon primarily because it was a nice monochrome look and had dual M.2 support (only the first slot runs at PCIe 3.0 x4 sadly due to the Ryzen 7/x370’s PCIe lane availability), but I wouldn’t recommend it. While it’s had some BIOS updates, it still doesn’t have the latest 4/10 AGESA update, and in general has had sluggish support and a few issues (my personal gripes: pokey POSTing, no last-good/soft-CMOS reset). Personally, if I was buying a top-of-the-line board, I’d probably go with the Asus ROG CH6 – while it doesn’t have a second M.2 slot, you could put a PCIe adapter board on the last slot for that. As a bonus, there are a bajillion USB ports.
Instead of buying new GPUs, I just brought along a couple RX470 mining cards (sadly, these two weren’t running the past two months – that would have been $400-600 of missed earnings w/ the ETH run-up). I have them beavering away in the background right now while the system idles. (I am running the latest 17.4.2 drivers but with a BIOS signature check bypass).
One note during installation is that I had hard-lockup problems when installing from a 2015 Windows 10 stick – you’ll want to make a new one, w/ a 12/2016 ISO I didn’t have any problems.
While my VR development has been a bit sidetracked recently (one guess why), with the release of Nvidia’s Pascal 10-series mobile GPUs, I’ve been looking forward to abandoning my previous portable VR workstations and switching to a more traditional, relatively compact laptop.
I was originally most interested in the Aorus X3 Plus V6, but its release was so delayed, that I ended up preordering the Dell Alienware 13R3 when it was announced in November. Due to some discount hijinks (and the fact that it hadn’t shipped when it was scheduled to), I ended up reordering during the Black Friday sales. Delivery was originally scheduled for Dec 5, but got pushed back until the 22nd, by which time I was already out of the country, so I only this week finally had a chance to break it out and put it through its paces.
So, first, some positive things:
OLED screen – the main reason I decided I wanted the Alienware 13, despite the long wait and a few other concerns, was because of the screen. While only 60Hz, the OLED pixels really are glorious – sharp, contrasty, with 1ms switching and a ridiculous color gamut (104% AdobeRGB). It’s a capacitive multitouch screen (I don’t like to touch my screens but the one time I accidentally did, it seemed to work), and it’s mounted on a very solid hinge that doesn’t not wobble *at all* when typing. There is an achilles heel, but I’ll save that for the ‘cons’ section.
Good performance – If you order the 13R3 now, you’ll get the latest Kaby Lake (7-series) processor – mine was in the one-month window where the Skylake (6-series) shipped, however the performance difference is minimal (the Kaby Lake might have an ever so slightly better boost clock). I got mine with the highest specced i7-6700HQ that despite worries due to thermal problems in many early subreddit threads, ended up running fine. CPU stress tests clocked it at about 75-80C running at a boost clock of 3.1-3.2GHz, with even lower temperatures undervolting w/ Intel XTU. Sadly, when running benchmarks like Valley or gaming, the system still ended up being about 20% slower than my compact (stock) i7-4790K and GTX970 combo, but I doubt that any other GTX1060 based portable system would do much better (the GPU clock of the GTX1060 didn’t boost to 1900MHz, but stayed a reasonable 1600-1700MHz at around 80-85C while gaming).
The keyboard and trackpad were top notch, both in feel, and surprisingly, with the lighting on the trackpad (this is configurable, but by default the trackpad glows when in use and it’s actually pretty neat). The top plate is a soft touch plastic that is nice, although even over a couple days of light use, has started to acquire a few spots of sheen (ick, I know).
The Webcam has IR support for Tobii eye tracking (never used) and Windows Hello logins (which actually works great and is delightful)
Build quality is super solid – it’s built like a tank and it feels like you could definitely use it as a bludgeoning tool and then continue on your merry way. It’s also worth noting that swapping RAM (2 slots) or m.2 SSDs (2 slots) is a breeze – literally 5 phillips screws on the bottom, which is a big plus. The service manual is online, and overall, near complete disassembly looks like a breeze.
Due to the delays, I ended up calling Dell customer service a few times, and while not always completely helpful, I didn’t have to spend too long on hold and most of the time it felt like they were moving the ball along, so kudos for that.
OK, now with the cons, which includes some pretty serious stuff, sadly.
The screen – As I mentioned, one of the main reasons I picked the Alienware 13 over anything else was the OLED screen. And it really is glorious – as long as you are in complete darkness that is (that Achilles heel I mentioned). Even in indirect or low ambient light, the screen is basically a mirror, and if you like running dark text terminals like me (which would also be better for the OLED’s battery life), you will spend a lot of time staring at your own reflection. I really can’t fathom why someone would have such an otherwise awesome display and then put it behind such a glossy piece of glass.
Size and weight – While the build quality is admirable, and in theory I knew it was heavier than some of the alternatives, it’s not until you get it in your hands do you realize how bulky it really is. The computer itself is 2.5kg (5.5lb), and the power brick is another 0.8kg (1.7lb) on top of that. In total, you’re looking at almost 3.3kg (7.2lb) for the package. The sizing isn’t much better. It’s relatively thick at about 24-27mm (the other dimensions aren’t super small either), but worst of all, the center of the laptop is actually pointy, not flat. This makes propping the laptop up or using many laptop stands a non-starter. I don’t really know what Alienware was thinking with that design element.
Battery life – shorter battery life is something that I expected but in practice, turned out to be unacceptably low (much lower than reviews and claims I had seen) 84t92ao. In my unscientific rundown test of random web browsing and YouTube video watching, I got 3h 10m of use from a full charge before it shut down. This was in a darkened room and I don’t think I heard the fans turn on once mind you, so I don’t think it was stressing the system (also, w/ the OLED screen, you can’t disable Optimus, so I assume it wasn’t working out the dGPU even).
I paid a few bucks extra for the Killer 1535 (vs 1435) network card, which is supposed to have solved some of the older 1435’s connection problems. I also uninstalled the Killer Suite and reinstalled just the drivers, as that’s supposed to help as well. Alas, for whatever reason, the 1535 would drop connection (well, remain connected but time out on packets) about once a day. I have half a dozen devices running on my AC wifi without issues, so I’ll lay the blame on the card – it’s a relatively minor issue since it’d only be about $30 and 5 minutes to swap it out w/ an Intel wifi card, but I figure I’d mention it, while I’m piling on.
As I mentioned, there’s some minor throttling under gaming loads (1600-1700MHz on the GPU vs the 1900MHz max boost), although angling the laptop for better ventilation didn’t seem to improve things much. I also noticed some minor occassional graphics memory corruption in certain overlays in the game. Note: performance improved by about 10% after updating to the latest Nvidia WHQL drivers. There’s probably more I could have done to tweak out performance, but the screen, life, and bulk really killed my enthusiam
As you might expect, after a few days realizing some of the shortcomings, I’ve ended up deciding to return the system. I think for those that don’t fly/travel as much, or that don’t mind extremely glossy screens (there must be a lot of people like that, because I feel like Dell isn’t the only offender here), this might be a good fit. There really are a lot to admire here, and the reviews that I read/watched were mostly positive, so I don’t want to give the impression that this laptop is a total stinker.
For those looking at lightweight/portable VR capable workstations, your options are still sort of limited. I’ve ordered a Gigabyte Aero 14 that will hopefully address the worst issues I had with the Alienware 13 – it has a matte anti-glare IPS display (sadly w/ average color gamut), is 1.9kg (the 2.4kg weight with the power adapter is the same as the weight of the Alienware laptop by itself) and is almost 50% thinner. It also has a beefier 94Wh battery (Gigabyte has made claims for 10 hours of use, but honestly, I’d be happy if it could hit 5h of web browsing) and an Intel 8260 wifi card. Gentech is also offering a free liquid metal CPU+GPU repaste so I have some hopes that performance might actually be a bit better as well.
Here’s a couple video reviews of the Alienware 13:
(Just look at that glare in the Linus video – he doesn’t mention it at all)
Note, this is actually cheaper (and brighter) than the cheapest reasonable 60 Watt Incandescent A19s I could find. These are 2700K soft white, 60W, 630 lumens bulbs rated for 3500 hours (pretty good!) and cost $4.97 for a 2-pack ($2.49/bulb).
Before we calculate the power-inclusive cost, lets just total bulb replacement cost based on rated lifespan. The LED bulb is $1.99/10K hours, the incandescent is $7.10/10K hours, and the halogen is $9.17/10K hours. It’s somewhat surprising, that even if electricity were free/infinite and a non-issue, the LED bulb would still be over 3X cheaper than the incandescent now (post 2014 phaseout, prices will probably keep going up if you can even find standard incandescents).
Now, lets look at power over the same 10K hours using the 13 c/kWh average rate. The LED light bulb would use 85kWh, costing $11.05. The incandescent would use 600kWh, costing $78. And the halogen would use 430kWh, costing $55.90
In my 2006 post, my 110M household number stats was actually probably outdated/from previous years. In any case, Statista pegs 2015’s households at 124.59M, we can conservatively ballpark 2016 household numbers at 125M
We can now do a 1:1 update of the 2006 numbers. Replacing a single incandescent light-bulb per household with an LED bulb (probably a lot less common now), you would save approximately $9B over the lifetime of a single LED bulb
When looking at the power usage, say taking the 5 hours/evening usage from the old post (1826.25 hours/year), we end up at just under 12M MWh power savings, or about 1.5 1000MW nuclear power plants
Based on a ballpark distribution cost of $250M (assuming bulb+handing out bulbs = retail cost of the bulb), we end up at about $21/MWh saved, far cheaper than USD/MWh costs for any type of new power plant.
The LED bulbs linked above are CRI80 while the incandescents/halogens are CRI100. A high CRI (90, 93) LED bulb runs at about $4 or $5. Either way, they should in general be more pleasant than most CFLs from last decade.
GWB signed the EISA that had a low-efficiency incandescent phase-out (triggering the introduction of halogen alternatives, more about that here and on wikipedia)
Last week, GoPro announced its long-awaited drone, the GoPro Karma. The most compelling part is that it’s a modular system that includes the drone and a 3-axis gimbal that can be detached and used as a handheld. The product itself is super polished and includes a carrying backpack (w/ a chest mount no less) for a very reasonable price. For $1099, you get the system with a GoPro HERO5, which offers top-of-class image quality – honestly, GoPro’s promo vid, entirely filmed w/ GoPros, needs to be seen in high quality (youtube-dl -f 266+140 'https://www.youtube.com/watch?v=vlDzYIIOYmM') to be believed.
Today, DJI responded with the DJI Mavic, a foldable/portable drone that is smarter (it has all the obstacle avoidance and object tracking of the Phantom 4 and more) and incredibly small (folding into probably half the volume). It also has a (super-tiny) 3-axis gimbal and a camera that looks about on par w/ the HERO5 Session. It is $999 w/o a carrying bag/backpack, and neither the camera or stabilizer are modular (it also requires your smartphone to use for navigation vs the Karma’s AIO controller). Here’s the best hands on I saw today:
I’m definitely in the market for a travel drone, and while the Mavic’s stability is perhaps a question mark (UPDATE: looks good on Casey’s video), in almost every other way, it’s an obviously a better drone than the Karma, but if you are looking for more general (ground + air) filming, to get the same setup you’ll need to either pack your phone and spend another $300 for something like the DJI Osmo Mobile or for all-weather/action use, a stabilizer and an action camera (conservatively $500 total). As a do-everying solution, the GoPro Karma is compelling, even more-so when you consider that you can upgrade the camera separately. (While the HERO5 is w/o a doubt “the best GoPro ever,” honestly I was hoping that GoPro would end up offering an Ambarella H2 option for 4Kp60 10-bit capture and >60Mbps recording. Well, there’s always next generation.)
As of right now, I’ve preordered the Mavic, as the image quality is good enough, I don’t much care about the GoPro accessory or software ecosystem (both of which could be big pluses for some people), and I’m much more weight/volume than price sensitive. I sold some of the GPRO today that I bought a couple weeks ago to defray the costs (I’m actually still pretty bullish on GPRO and still long on it. Also, was pricey by the time I looked at it, but AMBA has been on a tear and maybe be the real winner from the drone wars.)
I’d like to start doing some more video in general, but I’m currently a bit undecided on the right mix of cameras. A m43 upgrade is almost certainly in the cards (the new Olympus EM1mk2 will do Cinema 4K 237Mbps recording w/ IBIS as well as a monstrous 15fps mechanical shutter for stills. The GH5 will do UHD 4Kp60 422 although details are sparse since it won’t be out until mid-2017), but I’m a bit unsure on what kind of stabilized (waterproof?) camera I’d want to go with, and whether there are any worthwhile stereo-180 or mono-360 cameras coming out (I’m tracking a bunch of options, just haven’t seen anything great yet).
UPDATE: The Karma does not fare well in this comparison:
UPDATE 2: Hmm, maybe not so terrible, the Hero 5 video quality is better and you’ll be able to upgrade the camera as well…
By my count Lensley is going into year 8 (jeez) but despite working with all kinds of image and video processing over the years, I only just finally got around to sitting down and grokking ffmpeg’s filter_complex this past week.
While it’s conceptuallysimple, the syntax documentation is rather dense and unintuitive. Web searches will let you find how to do any number of things, and you can hack your way for most stuff (in the past I’ve used h264_mp4toannexb, various intermediates, and concats to chain things together for example), but recently I found two writeups that were particularly useful in breaking down how -filter_complex works:
This Stack Overflow answer does a great job breaking down the exact construction of a relatively simple multi-stream complex filtergraph
One thing to note is that image2’s frame inputs remain stubbornly limiting and it’s tempting to simply do image manipulation externally to pass in, but doing it all in ffmpeg was at least 10x better performance than using ImageMagick for preprocessing. ffmpeg is just stupidly fast. I simply end up writing temporary symlinks and calling the ffmpeg bin, but you can also now conceivably use PyAV in Python which has just recently added filtergraph access to its libffmpeg/libav bindings.
Another recent “I probably should have sat down and figured this out years ago” moment was finally figuring out single-line multi-hops. I’ve been an “intermediate” level SSH user for a while now, figuring out SOCKS proxying and autossh-based tunneling a long while back. All our Lensley machines automatically reverse tunnel to our config servers and I have both ssh configs and various portmapping scripts to make connecting relatively easy.
However, my access (whether it be connections, port forwarding, or file copying) has always required an extra step for chaining until I stumbled on a post about multi-hop proxying. Now that particular description is confusing as hell, but the secret is in the -t part of the command. Here’s a much clearer description. So now I have a script that lets me connect to a reverse tunneled machine in one command that looks something like:
ssh -t server ssh -t localhost -p $port
Of course, there’s more than one way to skin a cat, and it looks like OpenSSH 7.3, released just a few days ago, has just added a ProxyJump command that makes life easier.
That script I wrote also tests first to see if a machine is available on the local network using curl (better than ping I think. netcat -z would be preferable but even w/ time-out set to 1, it always takes 5 seconds to time out):
It’s been a long while since I’ve spent much time thinking about file compression tools, but I was running out of space on one of my smaller SSD’s and the culprit was mainly lots of large multi-GB TIFs and PSDs that had accrued in our Lensley Dropbox.
My first thought was xz, since it compresses super well, but I was growing old waiting for files to compress and decided to switch. lrzip seemed like a good option, and performance was much better than xz, and threaded (and with multi-file support and a nice progress UI to boot), but I also happened on another compressor shoot-out page (if you’re interested in this topic, definitely read this, there’s a lot more testing and nice pros/cons list for the apps) which concluded that lbzip2 was a good choice.
I gave it a spin, and sure enough lbzip2 is great. With defaults, it loses a few percentage points when compressing vs lrzip, but it also compresses on average twice as fast, with the kicker being that its archives are fully cross-compatible with bzip2, so pretty much any system can access the archives by default. I’ll be aliasing bzip2 to lbzip2.
(If speed isn’t as important (batch compression) and for max compression of large files, lrzip is probably better. Although a lot of people have been using xz, even with multithreaded support, I still didn’t feel performance was great so for personal use, I’ll probably end up mostly using lbzip2 or lrzip from now on.)