Category: Tech

Thoughts on the Palm Pre, G2, and iPhone 3G

Posted on by lhl

Last week, I ended up with a Google Ion and a Palm Pre. I also loaded up the 3.0GM on the iPhone 3G. With active service on all of these and a whole bunch of devices on hand, I’ll be writing up some of my opinions in depth here. Instead of spamming multiple posts, I’ll just keep adding to this page (people on feed readers may want to check back in a week).

This isn’t necessarily intended as a full review, although if you’re primary question is whether the G2 or Pre are better than the iPhone, the answer is basically no. Android is very interesting from a developer perspective, and the G2 isn’t horrible, but it has many UI shortcomings that would make me sad if I had to carry it around as my daily device. The Pre on the other hand, is something I’d carry around (and in fact I’ll be switching to it as my primary phone), but in most regards for most people it is not better than iPhone 3.0 (OS features, responsiveness, battery life, applications). Which isn’t to say it’s much worse, but that if I were to give a recommendation, all other things being equal, I’d recommend someone go w/ a 3GS over a Pre. Given that statement, why am I switching? Here are my reasons (they may or may not apply to you):

  • In the cities I frequent (SF, LA, NYC), AT&T reception has been getting increasingly worse – a good quarter of my calls drop in SF, especially when I head downtown, and this past LA trip racked up a good 50% call failure rate across West LA, Culver City, Downtown, and Arcadia (this I’m sure is exacerbated when both ends are on iPhones). In comparison, on calls I’ve made this week w/ the G2 (T-Mobile) and Pre (Sprint), I’ve had no dropped calls at the same locations. Also, 3G data just doesn’t work in downtown SF and it seems, anytime there are lots of iPhones in an area. This is a problem that seems to be common enough, but as AT&T seems to not be acknowledging these problems and is unable or unwilling to fix them, and with the introduction of both more desirable and cheaper iPhones, I can only surmise that this will get worse. (somewhat related to that, I’ll also be saving $15/mo on my bill ($360 over 24 months), which is a pleasant bonus.)
  • I’ll be dedicating a post to this soon, but my moral outrage at Apple’s App approval process has been pretty palpable – it literally revulses me when I think too hard about it, and it’s managed to burn off a lot of the good will and regard that I feel for Apple over the past few months (not as much as when I had to admin a labful of OS9 Macs <tic> but…) Calling it evil is I think pretty accurate. This behavior is enabled by the fact that the iPhone is currently the only game in town for mobile developers. I think that everyone (including, in the long term, Apple) would benefit if this weren’t the case.
  • The Pre surpasses the minimum bar of general usability I have and there are some aspects of the Pre that are noticeably superior to the iPhone. I mentioned a couple things in an earlier post like the quick camera and the unified inbox. Others include the way notifications work, which is non-modal and doesn’t steal focus (NOTE to Android PMs [I address this to the PMs because it’s clear there must not be any UI designers with any say whatsoever]) and the thoughtful ways to go into airplane mode (as an option on power-off and in the menu when tapping on the upper right).
  • This is a biggie for me, but if you are also into poking around on your phone, the Pre is the most hacker friendly phone I’ve ever seen (yes, much more-so than open phones like the Android or OpenMoko, as you can easily modify the UI directly (since the apps are in JS) instead of having to recompile GTK/Java-based apps or flash hacked firmwares). With an unencrypted and very clean embedded Linux OS w/ a standard layout and easy root access (bonus style points for the use of the Konami code to enable it), I have a feeling we’ll be seeing lots really interesting stuff coming out of the homebrew scene soon. Even better, the initial Palm response to this hacking has been benign, if not outright friendly. This is the right way to build up developer good will. Here’s my sandbox for poking around on that (and for projects that are outside the scope of the Mojo SDK).

The biggest caveat for those considering the Pre right now is the battery-life. For me, it’ll last for about 8-12 hours under very light usage, which is OK, but the bad part is that unlike my iPhone, the Pre sucks battery life at an alarmingly regular rate while idling. Whereas I can leave my iPhone unplugged overnight and maybe see a 1/4-1/3 drop, doing that w/ the Pre will leave you with a dead phone. The battery is still new, so it might get better with conditioning and I’ve heard that there’s an update that might be released soon to help address the battery issues, but as of right now… it’s an issue.

OK, so with that out of the way, here’s where I start nitpicking through things…

Physical Design

Both the G2 and the Pre are smaller (but thicker) than the iPhone. They both feel a bit lighter than the iPhone (the glass screen on the iPhone gives it a lot of that heft), but all three feel pretty good in the hand (it’s tossup between the G2 which is the least wide and fits better in my hand and the Pre which is overall rounder).

The G2 has only one connector a combination mini-USB/custom HTC adapter on the bottom (no 3.5mm headphone jack; there wasn’t an adapter in my box – I assume the retail versions will have one, but it’s still rather ridiculous) and a volume rocker on the side. On the front, there’s a trackball and 6 buttons (home, menu, back, search, call, and end/power). There are two LEDs on each side of the earpiece speaker near the top – they don’t seem to blink for notifications, just when charging or low on battery.

The lack of a hardware ringer switch is one of my pet peeves. In fact, with the G2, there’s actually no way to go into silent or vibrate mode without pulling it out of your pocket and unlocking the phone and then silencing it, as the volume keys are locked and don’t do anything otherwise. Personally, I consider this a pretty big minus, as the ability to easily go into and confirm that you’re in silent mode without pulling your phone out is something that I use all the time (meetings, movies, anywhere you’re around company and trying to be inconspicuous).

When closed, the Pre’s button layout pretty much matches the iPhone’s. The ringer switch is there of course (with a [too weak] vibrate to confirm), and the power button is on the top-left corner, which is convenient for handling calls (one click to dismiss, two to send to voice mail). My big nitpick for the buttons is that while like the iPhone, the two-step wake process involves a button-click and a gesture, in this case, the center-button doesn’t work, just the power button. Now, this may just be muscle memory (and I can to some degree understand why the center button doesn’t wake, as it protrudes as opposed to the iPhone’s better designed recessed center button), but, and this has happened a fair number of times, the power button is almost inaccessible when the slider is open, making waking/unlocking a complete pain in the ass when it’s open. It can be woken also by use of the slider, but…

The slider is another point where I’m not entirely happy with. Because of the Pre’s shape and surface, I’ve found it almost impossible to open single-handed (and sometimes difficult even with two hands). No one I’ve handed the phone to has had an easy time with it (or typically, even figuring out the sliding w/o prompting). I’ve just slapped on some Ghost Armor on, which actually seems to add just enough grip to make this significantly easier. I assume that the Touchstone back, w/ it’s soft-touch back would also make a big difference in that regard.

One thing that I mentioned earlier that I’m incredibly happy with is the Pre’s TRRS support on its 3.5mm jack – that means all your iPhone headsets will work (including the mic and the remote). I’ve noticed that there are sometimes issues with unpausing with the clicker, but as it seems to work flawlessly with the Pandora app, I’m going to chalk it up to Palm’s Music app, which is to put it charitably, a bit flaky.

Battery

While the battery-life of my iPhone was a definite step down from my Blackberry, in general use, both my original and my 3G have proved to be solid performers, generally managing to provide a full day (and more importantly, night’s) worth of charge. Under duress (conferences, etc.), the ubiquity of external batteries/standalone chargers has helped ameliorate those problems and the latest generation of case/battery combos are downright sexy. It’s funny to think that a couple years ago, a non-removable battery was a big deal, but now, it just isn’t…

The Pre seems to have been conceived prior to that point. Yes, the battery is removable. But the flip side of having a battery that doesn’t last through the day… I’m not sure that’s quite the selling point they were hoping for. There’s apparently a 1350mAh third party replacement battery (same form factor), which is a start, but I think there will need to be some software tweaking that focuses on some smart and aggressive power management to really fix this problem. (It’s by far the biggest weakness of the Pre I’ve found so far).

The G2 on the other hand has been a pleasant surprise (especially w/ the issues the G1 has had) and seems to have great battery life. Even during the first day with very heavy downloading (installing dozens of apps, poking around into everything) it had no problem making it through the day.

Keyboard

I type a lot on my phones (and have since getting my first Treo 600 and my original Danger Hiptop) and prefer QWERTY keyboards (I’ve switched back and forth between portrait and landscape, and now virtual vs physical). The keyboard on the Pre is pretty good (and laid out properly, w/ the @ and period on either side of the space) – I have no problems typing on it and the accuracy is much better than on the iPhone keyboard. That being said, I’m not sure it’s a huge speed boost and I find myself missing the corrections and the emoji international keyboards. The Android software keyboard on the G2 is also quite good, however, entering keyboard mode is sometimes awkward. Like the rest of the Android UI, there’s a certain lack of UI polish as things get pushed off (sometimes hidden) off-screen when the keyboard activates, but overall the Android keyboard worked surprisingly well.

Before I got the iPhone, I was as skeptical as anyone about the virtual keyboard, however after using it for a couple years, I’m now rather agnostic about it. What’s clear though is that the virtual keyboard has a huge advantage in flexibility, like the iPhone’s world class internationalization or for the still-as-of-yet largely untapped custom/app/domain-specific keyboard layouts.

OS Look and Feel

In my eyes, webOS is the first competitor to finally step up to the iPhone’s challenge in terms of presenting a well-thought out and genuinely delightful interface. It’s been interesting to watch this play out over the past couple years – not only did Apple catch the mobile manufacturers flat-footed, but we got to watch, Keystone Cops style as the various players tried to catch up, mostly pushing out half-baked, cargo cult touchscreen implementations that seemed to have missed most of the point.

The Pre’s general responsiveness is good – it’s gotten some dings on app-loading time, but it seems no worse (and sometimes much better) than responsiveness on my iPhone. One thing that the iPhone does much better for app-loading however is with affordances – the iPhone has a built in “zoom” that not only gives a 1s mask for the app, but also instantly lets the user know that the user was successful in launching, even if they are subsequently greeted with a blank screen, although the Mobile HIG gives recommendations (and the SDK code samples) for storing an app image snapshot. Android actually does this decently – apps seem to load much faster by default, but there is a swipe animation and a title bar replacement that gives you instant feedback on launching.

Another area where the iPhone still beats both the Pre and G2 is in responsiveness of the “desktop” – its combination of software physics and the physical surface feels just right (it may be an illusion, but 3.0 feels even zippier than 2.x) – very quick and responsive, super-slick, never laggy. The Pre is a close second – there is occasional lag with the launcher and it’s just a hair slower – and the plastic surface is surprisingly good. It really only suffers when you do a head-by-head comparison with the iPhone (so smooth…).

The G2 is (you might be beginning to notice a theme developing from here on out) far behind. The screen is supposed to be capacitive, but is laggier and draggier than the Pre, and doesn’t support multitouch. It doesn’t have any sort of physics at all, which, while boring is actually the least serious of my complaints. My biggest ire is reserved for the wonkiness of the drag/click interactions. The notifications bar (which is non-modal but which steals the focus from what you’re doing when there’s a new notification – STAB STAB STAB) seems to fill up quite quickly, but requires a flick with a certain distance/acceleration to pull down to dismiss (they can only be cleared en-masse). On the bottom, there’s a tab for the (alphabetically ordered only) launcher that reacts similarly, but also can be clicked to expand. This is complicated of course by the fact that if you don’t click exactly, it registers as a drag and you get a useless 2px animated move. Which fresh-from Stanford grad thought that one up? Even worse, this same tab is used elsewhere in the interface (like in the dialer) but doesn’t respond to clicks at all – you’re left to wonder if you accidentally dragged until you realize that in this interface the exact same looking tab is drag-only and doesn’t respond to clicks at all. Thanks!

The most interesting bit of the Android desktop, the three-pane “desktop” unfortunately suffers from the fact that there is currently only one really great widget (the lock2’s Weather Widget) – I mean really, what am I going to do with a Twitter widget that shows the last (yes as in single) tweet from my stream? I ended up (over) filling up the rest of the desktop with icons of the apps I wanted to use in a sensible/findable organization. To add insult to injury, the desktop often slows down (or disappears completely!), even though it’s… pretty useless.

(On the desktop note, it’s unfortunate that the Pre like the iPhone doesn’t allow any way of customizing the lock screen or the “desktop” (which is empty when there are no cards) w/ a Dashboard or Today style screen – this is something that BB and WinMo have had forever, and something that the N97 (and future Android phones) are on the right track with.)

More scorn needs to be heaped on the G2 for it’s accelerometer – instead of having a regular rotation like any normal implementation, it instead darkens and blurs your screen (rendering it illegible) for a good second or two before flipping it. This makes every single accidental flip that much more annoying – if you find yourself forced to use a G2, do yourself a favor and go to Settings -> Display -> Orientation to turn that off. It’s that badly done. The Pre’s accelerometer behavior in contrast is quite impressive – both zippier and less touchy than my iPhone’s.

A note on boot times. The iPhone takes about 1m45s to boot up. The Pre is just a hair worse, about 1m55s. The G2 does better at about 1m05s – 1m15s (the fudge is because the launcher is a bit stuttery as widgets and other things initialize).

Search

While Palm has made a big deal of their “Universal Search” and the idea of “just typing” if you’re stuck, it turns out that iPhone 3.0’s search is actually better – not only slightly quicker and more responsive, but also more comprehensive. Apple’s search indexes notes, calendar items, and emails among other things (none of which webOS does currently). Here’s hoping Palm can follow through with their search framework.

One other thing to note is that there doesn’t appear to be any type of adaptive ranking mechanism for searches on the Pre. That is, no matter how many times you pick the “Flickr” contact when sending emails, it’ll never rise up in the results list right now. This is a major drag and something that the iPhone has done since launch.

Email

One of the things that the Pre blows away the iPhone on is handling multiple accounts. As I mentioned in my earlier article, right now, if there were new mail, we’re talking about 4 clicks to check mail for each account. 2 clicks just to get to the top level to check which account new mail has come into… It’s a right mess. The search in 3.0 is a welcome addition, although it looks like you can either do a global search in the main search interface (click the home button twice) or search within a folder when you’re in the Mail app.

Strangely enough, the Pre handles my GMail accounts (one GMail, four GMail for domains) better than the G2 – Android has a special “GMail” app that can only connect to the single “primary” Google account you hook your Android up to (this connects the Contacts and Calendar as well), but then you’re forced to use a separate “EMail” app for everything else. This is… stupid. It also is kludgy and not very pleasant. Overall, I’m much happier w/ how the Pre handles things. Setup is a little weird (you have to wait for it to fail automatically connecting before you getting a chance to manually change the settings), but after things are humming, it’s generally pleasant (with IMAP IDLE support). There are currently some EAS issues being worked out apparently, but I don’t have to deal w/ Exchange so that doesn’t impact me. One annoyance w/ Palm’s new email notifications is that a clicking through takes you into the message, but backing out takes you to the message’s Inbox, not the unified inbox. It’d be nice to control the default view there, and it seems there might be a better way of navigating (a drop down where the title/header bar is?), but it’s not a bad start.

Maps

There’s barely a day that goes by (well, those days I guess when I don’t step out the door) where I haven’t my iPhone’s Map application. If minutes were logged for application usage (that’d be nice, actually), I’d guess that it’d be by far the number one app. As such, I’m rather sensitive to its various foibles (somewhat stingy caching, the extra clicks for routing/re-routing from the current location, lack of reverse chronological listing of searches or the ability to juggle multiple points) and had high expectations for these other phones, especially Android.

Unfortunately, both the Pre and G2 fall short. The Pre’s GMaps are functional, but it seems to be a pure web-app (head to maps.google.com w/ the TeleNav, the same guys that developed the AT&T Navigator now available for the iPhone. The nice thing is that this comes free on the Pre (it’s a $10/mo extra on the iPhone) and it actually works pretty well (including integration w/ the address book). The speaker voice is a bit hard to hear, and it absolutely kills battery life (I couldn’t find a way to pause it or put in in the background while you have a route input), but it’s pretty neat if you drive a lot (the updating ETA is quite nice).

Browser

All three are WebKit-based browsers. You wouldn’t expect that much difference, so the big surprise for me was how horrible the Android browser is – it’s pretty much unusable. It doesn’t zoom well at all, so pretty much all layouts go off the screen, and it doesn’t support double-clicking to zoom to a DIV or reformat, so you’re forced to use the fixed zoom buttons (no pinch and spread of course). I was pretty shocked. It’s also the only one of the browsers to not support HTML5 data storage (it has support Gears storage instead – what a pain). The iPhone 3.0 browser is currently the only one of those browsers to support W3C geolocation (which Safari 4 and Firefox 3.5 both support).

The Pre browser has been fine – pretty zippy (slightly faster than 3.0 on the 3G, although that may be network dependent), with only a few rough edges (lack of zoom for field inputs, YouTube embeds don’t link to the YouTube app, and it doesn’t lock dragging, presumably because of the Apple patent). The only real rendering issues I have is with the iPhone version of Google Reader – it seems to occasionally get lost in the page when expanding some of the articles. There’s probably some weird height calculation/scrolling issues going on or something…

Photos

App Store

Development

Couple of New Toys: Google Ion (G2) and Palm Pre

Posted on by lhl

It’s been a busy couple months of intense traveling and juggling way too many projects, but I’m happy to report that I’m beginning to cull both of those to a somewhat manageable level.

New ToysIn the meantime, this past week I’ve picked up a couple new toys. A Google Ion (aka HTC Magic aka T-Mobile myTouch 3G (really T-Mobile?)) that I got care of Bradley Horowitz (I’ve started a GitHub project to at least have some sort of accountability there) and a Palm Pre that I just picked up today.

I figure that since I’ve been using an iPhone (original and then a 3G) for the past couple years and I have active service on all the devices at the moment, I’ll do an iPhone 3G vs Google Ion vs Palm Pre shoot-out after I get a chance playing with the latter a bit more (weekend project?).

There’s a lot I’ve been meaning to post about, so this blog will probably be very mobile-heavy over the next few weeks… In the meantime, some notes:

  • Android has by far the most impressive SDK – the kind of apps that you can do with full phone access are just leaps and bounds beyond what you can do with either the iPhone SDK or with the Mojo SDK. The latter I’m bound by NDA on, but there’s enough public stuff to talk about. I’ll be doing a post soon (also where I’ll rant at length about Apple’s app approval process, and talk about marketshare). That being said, having played w/ almost 20 Android transit apps, I got to say, there’s definitely a real big gap there.
  • I’ve only seen one report of it, which I find rather amazing considering the number of reviews I’ve read, but the Palm Pre does indeed support have TRRS connector support, which include supporting the microphone as well as clicking for answering, pausing, and double-clicking for the next track. This works not just for the music player, but also for Pandora (and presumably other third party apps using the WebOS audio services/media extensions. This rocks, as I have a few hundred bucks worth of Ultimate Buds and am unconvinced on the audio quality that AD2P/AVRCP controllers like the Jabra BT3030 offer (not to mention the battery life issues both for the controller and the phone).
  • The Google Ion has a much better image quality (with auto-focus), but the Palm Pre gets props for both having the fastest shot-to-shot (and the least shutter lag) of any camera phone I’ve ever used. It’s actually faster than my Samsung point and shoot. And since you can leave it open in a card (the live display gets paused) where switching is almost instantaneous, I expect to have a lot less missed shots when spontaneous antics occur. Hopefully it’ll do video as well as it does stills…

  • OCD-leaning people beware, these phones are fingerprint magnets. Amazingly, the Ion and the Pre are both worse than the iPhone in that regard. They both have decent hand-feel, with the Ion fitting a bit better in my hand (a little longer, but less wide). Maybe I just haven’t figure out out the slider yet, but that’s probably the most awkward thing for me right now – there’s just nothing to push against to open it. Oh, also, prying off the micro-USB cap took a real leap of faith. I can see how people have snapped it right off. That’s basically what it feels like.
  • One software note (many more forthcoming) – is that I’m really digging the Pre Mail app’s combined inbox. IMAP IDLE w/ Gmail accounts works great, even better than on Android, which only supports a single GMail account (with everything else in a separate Email app). While I wish there were some improvements (like coloring or some other way of marking which email was going to what account) and it’s nowhere near as mature as the Blackberry inbox, this is still a huge improvement over email on my iPhone – w/ 7 email accounts, it takes 2 clicks to look at my accounts and 4 clicks to get in and out of each inbox (28 clicks total if I have new mail in each account).

Feel free to leave any specific questions and I’ll try to answer, I’m planning on doing a comparison w/ timings and end-user notes as well as one specifically on development and other thoughts. I’ll try to not get too obsessed, since there’s lots of actual work that I really should be doing instead, but at the same time, I have a fair amount to say, and I’ve used just about every smart-phone platform at one time or another and have a decent number of current OS’s to compare these to (I should probably dig out an old Treo and an N-Series to complete the set).

Some Notes on Distributed Key Stores

Posted on by lhl

Last week I ended up building a distributed keystore for a client. That wasn’t my original intention, but after doing testing on just about every project out there, it turned out to be the best (only?) solution for our needs.

Specifically, a production environment handling at least 100M items with an accelerating growth curve, very low latency retrievals, and the ability to handle 100s of inserts/s w/ variable-sized data (avg 1K, but up in many cases well beyond) … on EC2 hardware. The previous system had been using S3 (since SDB is limited to 1K values) – err, the lesson there, BTW is don’t do that.

So, these requirements are decent – something that actually requires a distributed system, but something that shouldn’t be beyond what can be handled by a few nodes. My assumption was that I’d actually just be doing some load testing and documenting installation on the keystore the client picked out, and that would be that. This was not the case.

I’m still catching up on a number of other projects, so I don’t have a great deal of time to do a formal writeup, hoewver, the work I’ve done may be useful for those who might actually need to implement a production keystore.

Some other recent useful starting points may be Richard Jones’ Anti-RDBMS roundup and Bob Ippolito’s Drop ACID and think about data Pycon talk.

  • MySQL – while the BDB backend is being phased out, MySQL is a good baseline. With my testing, on a single m1.large, I was able to store 20M items within one table at 400 inserts/s (with key indexes). Key retrievals were decently fast but sometimes variable. There are very large production keystores are being run on MySQL setups. Friendfeed has an interesting writeup of something they’re doing, and I have it on good authority that there are others running very big key stores w/ very simple distribution schemes (simple hashing into smaller table buckets). If you can’t beat this, you should probably take your ball and go home.
  • Project Voldemort – Voldemort has a lot of velocity, and seems to be the de facto recommendation for distributed keystores. A friend had used this recently on a similar-scale (read-only) project, and this was what I spent the majority of my time initially working with. However, some issues…
    • Single node local testing was quite fast – 1000+ inserts/s, however, once run in a distributed setup, it was much slower. After about 50M insertions, a multinode cluster was running at <150 inserts/s. This… was bad and led me to ultimately abandon Voldemort, although there were other issues…
    • There is currently only a partially complete Python client. I added persistent connections in as well as client-side routing w/ the RouteToAll strategy, but well, see above
    • Embedded in the previous statement is something worth mentioning – server-side routing currently doesn’t exist.
    • While I’m mentioning important things that don’t exist, there is currently no way to rebalance or migrate partitions, either online, or, as far as I could tell, even offline. This puts a damper on things, no?
    • As a Dynamo implementation, a VectorClock (automatic versioning) is used – this is potentially a good thing for a large distributed infrastructure, but without the ability to add nodes or rebalance, it means that for a write-heavy load, it would lead to huge growth with no way for cleanup of old/unused items (this of course, also is not implemented)
  • LightCloud – this is a simple layer on top of Tokyo Tyrant but the use of two hash rings was a bit confusing and the lack of production usage beyond by the author (on a whopping 2 machines containing “millions” of items) didn’t exactly inspire confidence. Another problem was that it’s setup was predicated on using master-master replication which requires update-logs to be turned on (again, storing all updates == bad for my use case). This was of course, discovered rooting through the source code, as the documentation (including basic setup or recommendations for # of lookup & storage nodes, etc is nonexistent). The actual manager itself was pretty weak, requiring setup and management on a per-machine basis. I just couldn’t really figure out how it was useful.
  • There were a number of projects that I tried, including Cassandra (actually has some life to it now, lots of checkins recently), Dynomite and Hypertable that I tried and could not get compiled and or set up – my rule of thumb is that if I’m not smart enough to get it up and running without a problem, the chances that I’ll be able to keep it running w/o problems are pretty much nil.
  • There were a number of other projects that were unsuitable due to non-distributed nature or other issues like lack of durable storage or general skeeviness and so were dismissed out of hand, like Scalaris (no storage), memcachedb (not distributed, weird issues/skeeviness, issues compiling) and redis (quite interesting but way too alpha). Oh, although not in consideration at all because of previous testing with a much smaller data set, on the skeeviness factor, I’ll give CouchDB a special shout out for having a completely aspirational (read: vaporware) architectural post-it note on its homepage. Not cool, guys.
  • Also, there were one or two projects I didn’t touch because I had settled on a working approach (despite the sound of it, the timeline was super compressed – most of my testing was done in parallel with lots of EC2 test instances spun up (loading millions of nodes and watching for performance degradation just takes a long time no matter how you slice it). One was MongoDB, a promising document-based store, although I’d wait until the auto-sharding bits get released to see how it really works. The other was Flare, another Japanese project that sort of scares me. My eyes sort of glazed over while looking at the setup tutorial (although having a detailed doc was definitely a pleasant step up). Again, I’d finished working on my solution by then, but the release notes also gave me a chuckle:

    released 1.0.8 (very stable)

    • fixed random infinite loop and segfault under heavy load

OK, so enough with all that, What did I end up with you might ask? Well, while going through all this half-baked crap, what I did find that impressed me (a lot), was Tokyo Cabinet and its network server, Tokyo Tyrant. Here was something fast, mature, and very well documented with multiple mature language bindings. Testing performance showed that storage-size/item was 1/4 of Voldemort’s, and actually 1/2 of actual size (Tokyo Cabinet comes with built-in ZLIB deflation).

Additionally, Tokyo Tyrant came with built-in threading, and I was able to push 1600+ inserts/s (5 threads) over the network without breaking a sweat. With a large enough bucket size, it promised to average O(1) lookups and the memory footprint was tiny.

So, it turns out the easiest thing to do was just throw up a thin layer to consistently hash the keys across a set of nodes (starting out with 8 nodes w/ a bucket-size of 40M – which means O(1) access on 80% of keys at 160M items). There’s a fair amount of headroom – I/O bottlenecks can be balanced out with more dedicated EC2 instances/EBS volumes, and the eventual need to add more nodes shouldn’t be too painful (i.e. adding nodes and either backfilling the 1/n items or adding inline moves).

There are some issues (an issue w/ hanging on idle sockets) but current gets are at about 1.2-3ms across the network (ping is about 1ms) and it seems to otherwise be doing OK.

Anyway, if you made it this far, the takeaways:

  1. The distributed stores out there is currently pretty half-baked at best right now. Your comfort-level running in prod may vary, but for most sane people, I doubt you’d want to.
  2. If you’re dealing w/ a reasonable number of items (<50M), Tokyo Tyrant is crazy fast. If you're looking for a known, MySQL is probably an acceptable solution.
  3. Don’t believe the hype. There’s a lot of talk, but I didn’t find any public project that came close to the (implied?) promise of tossing nodes in and having it figure things out.
  4. Based on the maturity of projects out there, you could write your own in less than a day. It’ll perform as well and at least when it breaks, you’ll be more fond of it. Alternatively, you could go on the conference circuit and talk about how awesome your half-baked distributed keystore is.

UPDATE: I’d be remiss if I didn’t stress that you should know your requirements and do your own testing. Any numbers I toss around are very specific to the hardware and (more importantly) the data set. Furthermore, most of these projects are moving at a fast clip so this may be out of date soon.

And, when you do your testing, publish the results – there’s almost nothing out there currently so additional data points would be a big help for everyone.

Virgin America’s Crappy Online User Experience

Posted on by lhl

These days I mostly prefer to fly on Virgin America. Their flight experience is a huge step above most of the other domestic carriers (friendly service, decent seats, regular non-prison inmate faucets, etc.) and touches like plugs in every seat, a good entertainment system (although there’s also a huge unfinished post about improving that), and now wifi, all at a competitive price makes it pretty much a no-brainer for me.

So, it’s always been a little surprising that for an airline with such a strong focus on branding and flight experience that seems targeted at people like me would have such a bad online experience.

I’m actually not going to bitch too much about the website (you know, about how it’s slow, has weird bookmark-unfriendly urls with weird sessions, is much too dependent on Flash with lots of weird interactions where it consistently takes me multiple times to log in because it’s login form doesn’t tab properly, etc). but rather to focus something that happened to me today that should have been a good thing.

I had a 2PM-ish flight back home today. At 11:30AM, an email gets sent to me from telling “Virgin America Guest Services” about an “Important Schedule Change Notification”:

Your flight has been impacted by a schedule change which may result in the departure time of your flight being earlier than previously scheduled.

That’s actually great – well, certainly better to be notified as soon as possibly than not to find out at all. And besides being good customer service, I’m sure it’s good on VA’s end if they can reduce the amount of shuffled seating that kind of schedule change might cause. However, it continues:

We’d encourage you to login to the Check-In / Travel Manager section of our website at virginamerica.com to view your current itinerary. You’ll need your elevate login information or your confirmation code (see below) and your last name to access your itinerary. If you have any questions regarding the new time please contact our Reservations call center at 1.877.FLY.VIRGIN (1.877.359.8474) between the hours of 3:30am – 11:30pm PST. You may already be aware of the new departure time and will not need to take any action at this time.

Now, this is cut and pasted directly from the email. It is an HTML email, but it doesn’t include even a link to the site, not to mention a link to the flight information. This of course is made doubly frustrating by the fact that it is a personalized email that includes my name, address, and confirmation number. Now, I’m not a rocket scientist, couldn’t they just save a step and include the flight information and what changed? If for some reason they couldn’t, why wouldn’t they include a direct link to that information? That’s all before you try to load the VA site on your phone. (Which works, barely, on my iPhone. Good luck with that if you don’t have 3G or WebKit.)

It seems that VA would actually save money if they could streamline this, since as it is, they probably get a lot of people calling rather than looking at the email and finding out what they need.

Since I like VA, the next step for me was replying and letting them know that it’d be great if they could include the information, a link or something mobile friendly. Unfortunately, once I got home, I saw that it was sent to a no-reply email address (bounced!). There’s no other contact VA from the email, unless you want to spend time on the call center, which isn’t a good use of anyone’s time.

Well, since I really do like VA (have I mentioned it’s incredibly easy to standby on an earlier flight?), I decide to go to the website and contact them… and after writing out my brief issues with 4 bullet points, it turns out there’s a 1024 character limit (yes, that’s 7 tweets and no dynamic character counter).

At this point, I probably should have given up, but I’m a sucker for sunk costs, so I went to look for an online character counter and started shaving off characters and doing some txt squeezing. In the end, they got my “feedback,” but it did get me thinking about this whole chain of events, and about how lots of these little bad UX decisions can compound to ultimately burn good will really quickly (and how difficult this sort of thing is to measure).

Now, I don’t think that this had a particularly big effect on my feelings about VA getting me from point A to point B decently, however it’s interesting to me when I compare say their level of quality/attention to detail for things like their safety video (the best I’ve seen) vs their online/digital UX.

From my perspective, I also think that there’s a pretty strong business case, and at least from some of these, ROI is calculable (ie, bucket-testing call % or missed flight percentage if you A/B test variations of the initial email), but for most of the rest of it, it’s not. To some degree, I also wonder whether a company like VA (or almost any company) really values how much of their UX and ultimately, (marketing, customer service, and brand) is dictated/deeply impacted by their online experience. They must have the numbers on what percent of their sales come through the website and what percentage of them are subscribed to email or use the mobile web.

Anyway, enough rambling. Now I’m just putting off all the work I need to do before my next flight…

What I’ve Been Up To Lately

Posted on by lhl

Since the beginning of the year, I’ve been spending most of my waking hours working on a new project with an old friend. It’s still a bit of a work in progress, but we’ll be at ETech this week and at events at SXSWi and SXSWm the week after, so what better time then now for a long rambly blog post introducing the Lensley Automatic.

Our new photobooth
hello.
A couple years ago, Jaime decided to build a photobooth (and with no prior experience, headed off to Home Depot…) and it’s been percolating along since. We’ve done events at the X-Games, the US Open, and with clients like Nike, Adidas, Diesel, Fuel.TV, Fuse.TV, MTV, etc. Towards the end of last year, after returning from a several month long interruption working on the Obama campaign (that worked out OK, btw :), we decided that it was time to take it to the next level.

It’s been an incredibly busy past few months, but what we’ve ended up with I think is something pretty unique (with a lot of potential). We have a new and improved enclosure (although, admittedly, a new version is already cooking), and more interestingly (well, it certainly took a lot more of my time) our own custom software for the booth, visualizations, and network interaction, giving us the ability to completely customize the printed output, the booth user experience, and the digital followup. For a start, we’ll be tweeting and posting photos to flickr w/ autotagging by way of RFID (fingers crossed on that!) at ETech. Just the first of the cool things we have planned.

And, while learning Cocoa hasn’t been all roses, it has been a great deal of fun working on a project that touches on hardware, visualization, photography, events, and the social web (and soon, video and mobile) – it’s a big cross section of “things I’m interested in.” Plus, all the joys of starting a small business (that’s half facetious, but also half genuine). Sure the timing might not be ideal, but all in all, it’s been a great experience in terms of stretching out some different muscles after being a bit cooped up. And well, there’s no time like the present to do your own thing.

Oh, if you’ve seen me in person in the past couple months (not likely!) and I’ve been more scatterbrained than usual (or have been responding in a zombie-like fashion), now you know why. (Not helped by the fact that for whatever reason, I spent a good few weeks of development time on a 4pm-10am schedule.)

Infrastructure for Modern Web Sites

Posted on by lhl

One of the things that I did when I wrapping up at Yahoo! was to begin to take a look at the current state of web frameworks. I ended up picking Django, but I have to say, I was disappointed with the state of what’s out there. Friends will have heard me bemoaning this sad state of affairs – that while Rails and Django might make CRUD easier, that the ORMs weren’t suitable for scaling beyond “toy” sizes, and that more importantly, they didn’t seem to address almost any of the pain points of building and maintaining a modern website.

A couple recent posts, most notably Krow’s Scaling, Systems Required list, but also Tom Kleinpeter’s post asking Where Are the AB Testing Frameworks? reminded me that I had made my own list. I was originally going to start working on these, but since I’ve now been side-tracked by a few projects, I thought I’d put it out there before it gets too completely irrelevant.

I’ve split this into two sections. The first I call “below the line,” which are more system level (some things straddle the line):

  • API Metering
  • Backups & Snapshots
  • Counters
  • Cloud/Cluster Management Tools
    • Instrumentation/Monitoring (Ganglia, Nagios)
    • Failover
    • Node addition/removal and hashing
    • Autoscaling for cloud resources
  • CSRF/XSS Protection
  • Data Retention/Archival
  • Deployment Tools
    • Multiple Devs, Staging, Prod
    • Data model upgrades
    • Rolling deployments
    • Multiple versions (selective beta)
    • Bucket Testing
    • Rollbacks
    • CDN Management
  • Distributed File Storage
  • Distributed Log storage, analysis
  • Graphing
  • HTTP Caching
  • Input/Output Filtering
  • Memory Caching
  • Non-relational Key Stores
  • Rate Limiting
  • Relational Storage
  • Queues
  • Rate Limiting
  • Real-time messaging (XMPP)
  • Search
    • Ranging
    • Geo
  • Sharding
  • Smart Caching
    • dirty-table management

The second section, which I call “above the line” are common application level components that typically depend on one or more of the components above. There are of course a huge list of features for any component, but I’ve highlighted some that either aren’t commonly implemented or are particularly important:

  • AuthX (AuthN + AuthZ)
    • Capabilities
    • Multifactor Auth
    • Rate Limiting
    • Signup
    • OpenID
    • OAuth
    • External import
  • Groups
  • Invites
  • Lists
  • Notifications
    • Spam filtering
    • Multi-protocol routing
    • Fine-grained controls/rules
  • Presence
  • Social Activity Log (Newsfeed)
    • Filtering
  • Social Model
    • Connectivity (uni/bidi)
    • Privacy (private, reciprocal, public)
    • Views
    • Traversal
  • Social Object
    • Privacy, Social Scoping
    • Voting
    • Sharing
    • Publishing
    • Comments
    • Favoriting
    • Social editing
    • Permissions
  • Tagging
    • Combinations
    • Relatedness
  • User
    • Achievements/Awards
    • Activity Log
    • External User ID Mapping
    • Permissions (see AuthX)
    • Deletion/Archival
    • Flagging
    • Direct Messaging
    • User Cards

This list is by no means complete, but maybe a good starting point. I’d be interested to hear what other people have had to build/would most miss if they had to start anew.

(What seems the biggest shame to me is that everyone is currently rebuilding this stuff over and over again and rationalizing it as some sort of secret sauce competitive advantage when it’s really infrastructure – stuff that really should be standardized so you can actually get around to doing the new and interesting stuff.)

Update: For those of you who feel the urge to comment about not needing this functionality: if existing frameworks work for you, that’s great. Also, if you’re not building a site that provides a service to users and have or are planning on being able to grow it, then you’ve likely not faced these pain points. Feel free to move along.

Now, I would like to hear from others working on similar problems, although I understand that most of those people remain under the corporate veil where this sort of information remains “competitive advantage.” Hopefully putting this list out there helps people realize that everyone’s building the same stuff over and over again (to varying levels of quality).

On Application Development

Posted on by lhl

The other day, Jeff Atwood posted a piece entitled A Scripter at Heart that distinguished programming vs scripting. Simon Willison had a strong (negative) reaction to that, and proposed distinguishing by the term “dynamic languages”. Yesterday Matt Biddulph posted a bit about some of his experiences as a web developer working with Objective-C and the iPhone (some more discussion), and since I’ve been doing something similar this month, I thought I’d throw in my 2-cents (my experience so far has differed from Matt’s), since it also relates to how I veiw the divide of two very different types of programming (systems vs application?).

To preface, like Matt, my background is also primarily as a web developer, although not exclusively – I’ve written my share of Lingo, Java Applets, OpenGL, Shake scripting, Max/MSP and Processing and other stuff. These days I hang my “expertise” hat on web architecture and systems, but I’ve done a fair amount of just about everything on the web side of things including some lower level things like working on Apache modules.

This isn’t to brag (you’ll note no accomplishments of merit mentioned above :), but simply to give some context of where I’m coming from. Learning Cocoa has been interesting. Of course, first and foremost, there’s the unique feeling of being a newbie again – that awful confusion, but also the excitement and then that somewhat retroactively forgetful feeling of incomprehension at not understanding how something works…

This learning phase may have maybe been more painful that it could or should have been. “Learning Cocoa” encompasses, not just a language (Objective-C) tied intimately to multiple very large sets of libraries (collectively Cocoa, but also CoreFoundation, AppKit, and in my case Quartz, Core Image and CoreAnimation as well as an inscrutable third party API), but also XCode and Interface Builder, each with a myriad number of settings, plists, etc.

While I think that a further discussion of the total lack of context and the bits and pieces of documentation/tutorials that did help me get my bearings may be the topic of another post, I did want to mention that the Apple Developer Documentation did not help me as I would have hoped in terms of orientating myself.

Some more observations:

  • It’s sort amazing how much more work seems to go into accomplishing very little, and how your ambitions scale along with that. I’ve spent more time working on looping some animations and making sure it doesn’t leak memory for example than say the Event SRP, or heck, the entire offline-task system on MyBO. Maybe it’s just my experience so far (biased say with spending a solid week fighting a certain third party SDK while learning the fundamentals), but I can see now why desktop apps haven’t seem to evolved as quickly as web services have. There’s just a lot of slog involved.
  • Note: PyObjc doesn’t make things easier – it’s just … hideous
  • Although… it would avoid Objective-C 2.0’s ridiculous memory handling – there’s garbage collection on the RunLoop, but only in some cases (for explicitly init’d, alloc’d and retain’d objects) but the AutoRelease doesn’t happen in threads, which by the way NSTimer launches, so make some subpools, but be sure not to over-CFRelease lest you cause an ecxeption (and crash) down the line, but good luck w/ MallocDebug if you missed anything and need to track it down… Don’t I have better things to do with my brain cells?
  • Casting through contexts is just out of control. NSImage, CGImage, and CIImage? Really?
  • Get used to writing at least 10 LoC to do what seemingly should be a single easy action (or declaring something in at least two if not more files and sections). Coming from scripting languages, the amount of boiler plate is mind boggling
  • Also, as someone used to CPAN, PEAR, and PyPI, it’s also been interesting discovering how spoiled by the ease of third party libraries and how much less common and more effort it takes. Maybe I just haven’t gotten quite that far yet…
As a web developer, I’ve often complained about the crudity and lack of development and debugging tools, but having dipped my toe on the flip, I guess it’s tough all around. Application development seems to be dense, convoluted and, well, sometimes just plain masochistic.
It’s also interesting that for as many (and there are many) calls there are in the standard Frameworks, how equivalently difficult it seems to be to do anything that you *want* to do (this will be another near-future post where I talk at length about the current state of web “frameworks”).
But, who knows, maybe in a few months I’ll at this post and shake my head and wonder how I could ever be so confused.

Recently Reading

Posted on by lhl

The middle of last year was pretty much completely dominated by politics for me. Memeorandum replaced Techmeme as my starting page, and TPM and FiveThirtyEight were at the top of my reading list. Since then, my attention has started floating back. Here are some of my recent faves (blogs that have been intersecting well with some of my current interests):

  • aaronland – for whatever reason (having more spare time? 🙂 I noticed myself reading more of Aaron’s excellent (and lengthy) essays this year. (geo, maps, photos)
  • tecznotes – another one of the blogs that I’ve been following for a while that I’ve been digging a lot more – probably also has to do w/ being able to get back into doing cool stuff. Mike’s everyoneiknowisdoingawesomeshit tag seems apt to mention here (visualization, maps)
  • Duke Listens! – there are a couple music technology blogs I’ve been following, but this one, by Sun researcher Paul Lamere I think has been the most consistently interesting (music, recommendations)
  • SmugBlog: Don MacAskill – I’ve spent more than my share of time the past couple years thinking about scaling, and it was nice to find a blog/community of people talking about some of the nuts and bolts (mysql, hardware, scaling)
  • Perspectives – James Hamilton (AWS) has also been publishing some great stuff along those lines, mostly around data center efficiency (data center, hardware, scaling)

I also now have a pretty reliable stream of AV stimulation through my Vimeo channels and groups. Not that there’s any shortage of interesting things – it always amazes me when people talk about being bored online – attention continues to be what’s in short supply for me, even now being able to set my own schedule.

I’ve been cranking away for the past few weeks on a new project, but hopefully I’ll have a chance this week to catch up w/ some posts, including some of the stuff I’ve been working on.

New NSA Surveillance Revelations

Posted on by lhl

New information on targets for domestic surveillance on tonight’s Countdown with Keith Olbermann:

Visit msnbc.com for Breaking News, World News, and News about the Economy

Interestingly, so far, Google News is showing only 3 stories following up on this, which is surprisingly since the implications… are disturbing to say the least.

From Russell Tice’s Wikipedia entry:

In a press release issued by the National Security Whistleblowers Coalition on December 22, 2005, Tice explained the public aspect of his charges, stating that:

“As a Signals Intelligence (SIGINT) officer it is continually drilled into us that the very first law chiseled in the SIGINT equivalent of the Ten Commandments (USSID-18) is that Thou shall not spy on American persons without a court order from FISA. This law is continually drilled into each NSA intelligence officer throughout his or her career. The very people that lead the National Security Agency have violated this holy edict of SIGINT.”

…

In a letter dated January 10, 2006, Renee Seymour, Director of the NSA Special Access Programs Central Office, warned Tice that members of neither the House Permanent Select Committee on Intelligence, nor of the Senate Select Committee on Intelligence had clearance to receive the classified information about the SAP’s that Tice was prepared to provide.

…

On February 14, 2006, UPI reports Tice testified to the House Government Reform Subcommittee on National Security, Emerging Threats and International Relations that the Special Access Program might have violated millions of Americans’ Constitutional rights, but that neither the committee members nor the NSA inspector general had clearance to review the program.

There’s some more background at SourceWatch. I’ve got to simultaneously admire Tice’s patriotism and bravery, and also, well, hope for the best for him. Heck, I wonder if he’s been able to get another job since? Also, just thinking about the ramifications of how he, or really any citizen’s communications being completely monitored… just reflecting on that sort of information asymmetry is corrosive to any idea of how a democracy could function.

Open Sourcing and Improving the Citizen’s Briefing Book

Posted on by lhl

The formatting got a little messed up (no paragraphs!) for my posting, but I’ve left my 2-cents for the Obama Administration for my thoughts on implementing and improving future participatory online tools.

If you’re so inclined, give the posting a look (and vote or comment on what you think). The Citizen’s Briefing Book project closes at 6PM today, so if you want to put anything up, now’s probably the time to do it. (there are almost 50K entries so far – that’s a lot of internet rantings to sort through 🙂

Also, a copy of what I posted (w/ paragraphs):

Open Sourcing and Improving the Citizen’s Briefing Book

The Citizen’s Briefing Book was a great project and I’d like to commend everyone involved. I’m looking forward to seeing these experiments with participatory tools continue with the Administration at WhiteHouse.gov.

This is an area I’m particularly interested in, so my suggestions pertain to the meta-discussions that have popped up around improving the Briefing Book (voting bias, aggregating and surfacing related ideas and discussions probably being at the top of that list). While the suggestions themselves point to specific weaknesses in the current implementation, I believe that they more importantly highlight the larger opportunities of generating useful discussion, analysis and direct involvement in improving future participatory online tools.

I’m aware that most government IT is contracted out, but the development of these online tools should perhaps be an exception, both because of their strategic importance to the government and the people, but also for the more pragmatic reasons of their development intensiveness and the deep, fast-changing, and often esoteric expertise required. As talented as agencies like Reside, or Blue State Digital are, many of the most difficult challenges exceed the capabilities of any single group and would benefit from tapping into a much larger pool of motivated technologists.

While, MyBO (which, full disclosure, I worked on last year) and Change.gov were campaign or pseudo-governmental projects, as “government works,” the new WH.gov projects would provide a great, high-profile opportunity for the new Administration to embrace an open process, not just in publishing the source code, but by actively encouraging participation and engagement with distributed source control, open APIs and bulk data access, and dedicated discussion and feedback loops. Furthermore, it’s my belief (based on my observations and experiences within the high technology and the Internet) that creating a transparent and level playing field would also serve to encourage the best and brightest in industry when it comes to contributing infrastructure and other resources that would be required for any sort of serious online undertaking.

There are many talented people working on political tools, and many great third-party non-profits (Sunlight, Maplight, MySociety, to name a few) working on data transparency and other aspects of digital government, but when I look at the challenges facing the development and scaling (in both technical and social interaction terms) of what may eventually be the most transformative of new online democratic tools–those for radically distributed policy deliberation, agenda setting, and direct involvement–it seems to me that fostering an open approach would do much to spur development with tremendous benefits (and almost no additional cost) for all parties, not least of which would be the American people.

Notes on the development of the Citizen’s Briefing Book app: